Lessons Learned From Applying a Monitoring and Evaluation Framework to Economic, Social, and Other Health Impacts of the COVID-19 Pandemic

Abstract

Individual and community-level COVID-19 mitigation policies can have effects beyond direct COVID-19 health outcomes, including social, behavioral, and economic outcomes. These social, behavioral, and economic outcomes can extend beyond the pandemic period and have disparate effects on populations. Public Health–Seattle & King County (PHSKC) built on the Centers for Disease Control and Prevention’s community mitigation strategy framework to create a local project tracking near–real-time data to understand factors affected by mitigation approaches, inform decision-making, and monitor and evaluate community-level disparities during the pandemic. This case study describes the framework and lessons learned from PHSKC’s collation, use, and dissemination of local data from 20 data sources to guide community and public health decision-making. Social, behavioral, economic, and health indicators were regularly updated and disseminated through interactive dashboards and products that examined data in the context of applicable policies. Data disaggregated by demographic characteristics and geography highlighted inequities, but not all datasets contained the same details; local surveys or qualitative data were used to fill gaps. Project outcomes included informing city and county emergency response planning related to implementation of financial and food assistance programs. Key lessons learned included the need to (1) build on existing processes and use automated processes and (2) partner with other sectors to use nontraditional public health data for active dissemination and data disaggregation and for real-time data contextualized by policy changes. This project provided programs and communities with timely, reliable data to understand where to invest recovery funding. A similar framework could position other health departments to examine social and economic effects during future public health emergencies.

In January 2020, a Washington State resident was confirmed to be infected with SARS-CoV-2, the virus that causes COVID-19. ¹ In response, the Centers for Disease Control and Prevention (CDC) activated its Emergency Operations Center to support responding to what would become the COVID-19 pandemic. ² Because there was no cure, no known effective treatment, and no vaccine at the time, implementing nonpharmaceutical interventions (NPIs) was the primary method used to slow the spread of COVID-19. NPIs have a long and successful history of curtailing the spread of infectious diseases.^3,4 As SARS-CoV-2 spread across the United States, states and localities implemented or removed various NPIs, which included stay-at-home orders, restrictions on group gatherings, school closures, and closure of nonessential businesses. ⁵

Purpose

NPIs are effective at limiting the transmission of communicable diseases, but they can have additional effects on social, emotional, economic, and other physical and mental health outcomes beyond the direct health effects of COVID-19. ⁶ The ability to monitor these effects in near–real time is helpful to inform decision-making, understand indirect effects of NPIs, and monitor disparities and social determinants of health (SDH). Even after NPIs are relaxed, NPI-related effects can persist and have differential effects on populations. ⁶ Recognizing how NPI strategies are affecting populations can help policy makers consider how to use resources to ameliorate some of the disparities and/or work with communities to strengthen resilience in recovery. Monitoring can allow for resources to be directed where they are most needed, allow policy makers to consider changes to processes, and help communities communicate their needs.^7,8 We provide a brief overview of the monitoring and evaluation (M&E) framework and outcomes, with a focus on describing the lessons learned from applying this M&E framework that can inform the work of other health jurisdictions.

Methods

CDC developed an M&E logic model and approach designed to understand the social, emotional, and economic effects of NPI strategies that provided guidance, considerations, and technical assistance for state and local health departments, agencies, and partners. ⁹ The logic model used relevant, novel, current data to understand the effects of NPI strategies during and after their implementation; how the effects of NPI strategies varied by COVID-19 risk group, race and ethnicity, place, sex and gender, health status, socioeconomic status, sexual orientation, and disability status; and whether disparities changed during the pandemic and recovery period. ¹⁰ It included evaluating whether adjustments in NPIs or provision of additional supports addressed the adverse effects of NPIs. CDC reviewed our project, which we conducted consistent with applicable federal law and CDC policy. The CDC Institutional Review Board waived ethical review because the project was considered a nonresearch public health activity.

Working with the Washington State Department of Health and CDC, Public Health–Seattle & King County (PHSKC), the local health jurisdiction for King County, Washington, created a local M&E framework to monitor county changes in social, economic, and overall health and well-being due to the pandemic, beginning in April 2020 and extending through the public health emergency. ¹¹ In this article, we demonstrate examples of high-level outcomes; additional outcomes and details are described elsewhere. ¹¹ We also share lessons learned from implementation of this framework to inform the work of other public health jurisdictions.

PHSKC tracked NPIs and critical policy changes at the city, county, state, and federal levels, contextualizing social, economic, and overall health data by race and ethnicity, geography, and additional demographic characteristics with policy timelines. The initial intent for this 2-year funded project was to examine effects of the pandemic and evaluate distal effects of NPIs in mitigating transmission; however, the pandemic’s trajectory required more ongoing monitoring than planned.

The PHSKC M&E framework was designed to be replicable by other jurisdictions and used relevant, timely data that extended beyond traditional public health datasets; we collected publicly available and confidential data from 20 local data sources across other sectors (Table). PHSKC and the Washington State Department of Health implemented a mixed-mode (internet/telephone) multilanguage survey in August–September 2020 to collect additional information about effects of the pandemic on SDH among a convenience sample of Washington State adults. ¹³ PHSKC stratified data from these 20 sources (Table) by demographic and geographic elements to analyze, visualize, and release publicly available dashboards, briefs, and other products. ¹⁴ Subject matter experts at PHSKC and community partners reviewed data briefs for context. Where possible, PHSKC compared the latest available data with prepandemic data patterns and trends to contextualize policy implementation timelines to provide insights into changing patterns, which were visualized on the PHSKC website (Figure). ¹⁴

Table.

Data sources used in a local monitoring and evaluation (M&E) framework of economic, social, and other health effects of the COVID-19 pandemic in King County, Washington, 2020-2022 ^a

Data source	Content area	Coverage	Cost	Granularity	M&E access
Cell phone–based mobility a	Economic	National	No cost at the time; cost associated now	Aggregate	Newly acquired
US Census Bureau, Household Pulse Survey 12	Economic, social, overall health	National	No cost	Aggregate	Newly acquired
National Domestic Violence Hotline call data a	Overall health	National	No cost	Aggregate	Newly acquired
Washington State Employment Security Department a	Economic	State	No cost	Aggregate Line level	Newly acquired Requires data-sharing agreement
Washington State Employment Security Department a	Economic	State	No cost at the time	Aggregate	Newly acquired
Medicaid data a	Overall health	State	No cost	Line level	• Existing access • Requires data-sharing agreement
Syndromic surveillance a	Overall health	State	No cost	Line level	• Existing access; adapted for project • Requires data-sharing agreement
Local community health survey a	Economic, social, overall health	State	No cost	Line level	New resource
Child Protective Services data a	Overall health	State	No cost	Aggregate	Newly acquired
Adult Protective Services data a	Overall health	State	No cost	Aggregate	Newly acquired
State provisional death certificate data a	Overall health	State	No cost	Line level	• Newly acquired • Requires data-sharing agreement
Washington Poison Center calls a	Overall health	State	No cost	Aggregate	Newly acquired
Washington State Tobacco Quitline data a	Overall health	State	No cost	Aggregate	Newly acquired
Calls to behavioral health crisis line a	Overall health	Local	No cost	Aggregate	Newly acquired
Utility assistance program a	Economic	Local	No cost	Aggregate	Newly acquired
2-1-1 call data a	Economic	Local	Cost required	Line level	• Existing access; adapted for project • Requires data-sharing agreement
Technology broadband survey a	Social	Local	No cost	Aggregate	Newly acquired
Emergency medical services calls a	Overall health	Local	No cost	Aggregate	In-house newly acquired
Legal filing data from the King County Prosecuting Attorney’s Office and King County Department of Judicial Administration a	Overall health	Local	No cost	Aggregate	Newly acquired
Medical examiner office data a	Overall health	Local	No cost	Aggregate	Existing access

^aIndicates that data are unpublished. While some data sources shown in the table are national, publicly available data, other data are from national systems but are held by a state or local agency or are routine sources available from state or local agencies and, thus, are potentially available in other jurisdictions. Cost, data-sharing agreements, and data granularity can inform local availability of data and resulting capacity and resources in generalizing this framework to other public health jurisdictions.

Figure.

Example of economic, social, and overall health effects of the COVID-19 pandemic, King County, Washington, 2020-2022. Data on the weekly number of unemployment claims are combined with information on nonpharmaceutical intervention implementation and mitigation policies that illustrate the policy context for the data. Data source: Public Health–Seattle & King County. ¹⁴

PHSKC tracked the dates of inception and expiration for more than 100 federal, state, and local policies for NPIs and relief across 70 federal, state, and local websites using a word-search approach with regularly scheduled reviews. Examples included state expansion of individual access to unemployment benefits, expansion of health care access through telehealth and waivers of deductibles, expansion of remote service provision, protections against evictions, and business tax relief.

Outcomes

NPIs for COVID-19 had cross-cutting social, economic, and health effects. We observed racial and ethnic and geographic disparities across most indicators in King County. ¹⁴ Washington State and local jurisdictions issued strict stay-at-home orders with essential worker exceptions beginning on March 23, 2020. The number of unemployment claims rose immediately (Figure), and traffic volumes decreased (not shown), tracking in real time with ongoing NPI implementations and expirations.^15,16 Housing and food-related needs increased substantially, and fluctuations and disparities have persisted since the start of the pandemic. Food insufficiency was 3 to 5 times higher among families that expected a job loss or sustained a job loss than among families that did not expect or sustain a job loss, among families with lower incomes than among families with higher incomes, and among families in racial and ethnic groups (ie, Black, Hispanic, multiple race, American Indian/Alaska Native, or another race) than among the race group with the lowest reported food needs. Depending on the exact period of the data point, the families with the lowest reported food needs varied and were either Asian or White. ¹⁷ Housing concerns remained high ¹³ ; local data showed an initial increase in the number of telephone calls to 2-1-1, a health and human services referral hotline, concerning housing. In addition, the number of telephone calls increased when threats of federal or local eviction moratoria expirations were made. ¹¹ City and county policy makers extended local eviction moratoria past the federal expiration date, partially because of the knowledge of housing concerns and eviction prevention outreach.

Schools closed in March 2020 and were remote from April 2020 through the 2020-2021 school year. This closure affected 7.5% of households without internet access, including approximately 21 200 children without a computer in the household or broadband internet access, which was essential for remote schooling. ¹⁸ Compared with White residents, Black and Hispanic residents reported higher rates of limited access to high-speed internet and not having enough devices. ¹⁸ In response, local school districts, community-based organizations (CBOs), and local governments purchased and distributed internet hot spots and worked with internet providers to make no-cost internet available to income-eligible households. ¹⁸

Health insurance is often tied to employment, so when stay-at-home orders led to the closure of nonessential businesses, many employees lost employer-sponsored health insurance. ¹⁹ The percentage of working-age adults without health insurance almost doubled in King County after initial NPI implementation, from 6.9% in April 2020 to 11.8% in June 2020. ¹¹ Because Washington State expanded Medicaid eligibility as part of the Affordable Care Act, low-income adults can be eligible for Medicaid coverage. During the emergency declarations due to COVID-19, individuals were not disenrolled from Medicaid. The program implemented additional outreach and enrollment efforts in response to the pandemic. ¹¹

Mental and behavioral health remained of high concern throughout the pandemic, and several state and local task forces focused on understanding these data patterns and providing alerts to medical providers and communities when levels were significantly elevated. ²⁰

Lessons Learned

During a public health emergency, when resources and capacity are strained, an M&E framework can be challenging to execute but is an important component to understand the effects that occur beyond the disease outcome. Public health decision makers rely on timely, reliable data amid a pandemic to understand implementation and effectiveness of recovery dollars and programs. Partner programs and communities were able to use these data to communicate needed services or modifications, showing the effectiveness of this M&E framework. We learned several key lessons that can be applied to our future work and to future M&E activities.

Build on Existing Access, Create Automated Processes, and Document and Train as You Go

This project began by leveraging existing data sources in house, with a goal of finding data to fit the M&E framework. Workplans of existing full-time employees were shifted: 2.5 full-time employees were assigned across 8 epidemiologists and program/policy staff, allowing subject matter experts to focus on specific topics. This project used a recently launched countywide Tableau server that allowed for external data visualizations. Staff had expertise in data analysis, data visualization, and data dissemination. Weekly team meetings covered analyses, methodological concerns, new datasets, dashboard development, and user experience. As the pandemic continued, PHSKC expanded existing relationships and developed new relationships with external and internal data partners to eliminate data gaps or address needs expressed by community members and policy makers. For example, new data-sharing agreements/memoranda of understanding were developed, which expanded the team’s expertise across nontraditional public health datasets. Indicators were added as new data sources became available and/or archived depending on data relevance. Most public health jurisdictions may have similar data available (Table). ²¹ Jurisdictions that may have challenges immediately accessing some of these data could consider areal indices. ²² Given the rapid frequency of updates, particularly early in the pandemic, full-time–equivalent employees were cross-trained in data analyses and reviews, which kept the project on track when someone was assigned to other COVID-19 response activities.

Once the initial work of ingesting and analyzing the datasets was complete, developing detailed process documentation, generating iterative statistical code for processing, creating and using templates for analysis, using a framework for displaying the data in dashboards, and creating dashboard templates streamlined updates. Also, PHSKC received data in standard formats from data providers and used application programming interface calls (eg, Census Household Pulse Survey ¹¹ ) to import existing tables. Prior to the public release of data, PHSKC loaded data into a Tableau server development environment to review the data and visualizations for accuracy. Examples of framework documents can be shared upon request.

Disseminate Often Using Multiple Methods

The primary dissemination method was an interactive, online dashboard that showed trends, snapshots, and demographic differences and was updated weekly or monthly, depending on the frequency of the data source. Dashboards allow for sharing voluminous information in a visual format that may be more accessible than static reports to community members and policy makers. Where data were available, dashboards presented data at granular levels (eg, subcounty geography, race and ethnicity, age, sex and gender, sexual orientation), highlighting disparities and areas where additional resources may be needed. The dashboards contained snapshots and trends over time with a policy timeline to show how policies interplayed with the trends. A thoughtful dashboard design with granular data considers tradeoffs among accuracy, privacy, and frequency of data updates. As a best practice, PHSKC worked with communities and subject matter experts on messaging to ensure outputs that include disparities are not creating more harm or stigma and to validate whether quantitative findings are matching the community’s on-the-ground experiences.^23,24 Because of the breadth of the data shared, PHSKC rarely received data requests that could not be answered by using the existing dashboards.

A robust dissemination strategy helps reach individuals at varying levels of data literacy. This project used blog posts, social media, infographics, short topical reports, presentations, and a biweekly newsletter, and these companion products were useful for various data consumers, such as policy makers, health department staff, community members, and/or CBOs, particularly in an environment where there was a large media saturation about the topic. More than 4500 leadership, community, and CBO subscribers received newsletters with data highlights and links to the dashboards for additional information. Infographics or dashboard images were used by blogs, community outreach providing education, members of the community, and CBOs for applications to programs and services.

Partner With Other Sectors

Addressing SDH—housing, criminal–legal, transportation, food, internet, and employment—is critical for ensuring that public health decision makers can understand the wide-ranging effect of the pandemic and make evidence-based decisions about NPIs and recovery investments or programs. ²⁵ Many of these data have not been commonly used in public health systems but have high utility as early indicators of SDH effects. Data-sharing agreements and memoranda of understanding are often needed to use this information, but some data providers may share aggregate summaries that can be visualized to mitigate the need for a data-sharing agreement. The 2-way data street (sharing back with partners and validating findings with them) is also valuable for building relationships beyond the pandemic. Indicator data that show where certain communities or geographic areas are lagging in recovery or where disparities are increasing provide valuable insight for local, state, and federal policy makers. PHSKC shared findings and data products with policy makers and communities to inform program development and implementation in equity-based recovery. Partnering with other sectors may also involve collecting additional qualitative data that are commonly missing from administrative datasets. For example, PHSKC partnered with CBOs for the community of people living with a disability, as data on ability status are not routinely collected. ²³

Use Real-Time Data Changes Coupled With Policy Changes

We updated dashboards daily or weekly during the highest periods of NPI to ensure timely data; going forward, updates could be less frequent as mitigation measures are relaxed and indicators become less salient to the response. For example, King County ended all NPIs on June 30, 2021, after reaching a 70% vaccination rate for people aged ≥16 years. ²⁶ As stay-at-home orders ended, monitoring traffic volumes as an indicator of commuting and travel became less relevant and was no longer updated. Social mitigation policy changes provide context to each topic because some of these policy changes prevent negative outcomes (eg, unemployment as a sequela from business closures) that would be occurring if the policies were not in place. These M&E data were shared with the county’s eviction prevention program, guided decision-making in the distribution of millions of dollars in food assistance funding, and informed the county’s budget process. Housing authorities used the findings to understand and respond to their residents’ needs.

Because NPIs waxed and waned and overlapped across city, county, state, and federal jurisdictions, understanding these changes took a full-time–equivalent employee to ensure dates of the policies were accurately captured. When King County reinstituted a partial stay-at-home order, a corresponding uptick in unemployment filings occurred, and having the context of the partial NPI helped CBOs and policy makers understand what was happening. Surveillance monitoring of policy inception, actions, and end dates provides important context and is a novel approach for public health departments. Future approaches such as developing natural language processing algorithms may be a way to decrease the amount of manual review time while increasing rigor in documenting changes.

Disaggregated Data Are Needed to Assess Health Disparities

Disaggregation by demographic characteristics is critical in ensuring responses to M&E findings that support equity rather than inadvertently widen existing disparities. ²⁷ Disaggregation includes detailed disaggregation, such as detailed race categories, in addition to demographic characteristics such as age, sex, gender, geography, socioeconomic status, ability status, and any other characteristics that may be uniquely affected by NPIs, such as employment and occupation. ¹⁶ Not all datasets collect the same demographic or geographic detail, which can limit the usability of the data and may leave out some communities. For example, most administrative datasets do not have information on sexual orientation, gender identity, occupation, or ability status or may have limited breakdowns on race and ethnicity data. It may be necessary to conduct other local surveys or gather primary qualitative data to understand the effects of NPIs on these populations. For the datasets that did collect this information, disaggregated data provided opportunities to examine effects, including the intersectionality of demographic characteristics. Ongoing, regular examination of disparities by demographic characteristics is important for monitoring the short-term and long-term (including generational) effects of NPIs. ²⁸

A multifaceted M&E approach—blending data across various data siloes for public health monitoring—provided actionable insights to understand how NPIs affected social, economic, and overall health outcomes across various cultural and geographic populations. Tying timely data together with real-time policy decisions provided the opportunity to move beyond anecdotal stories of community effects. This generalizable M&E framework and lessons learned expanded the ability to examine SDH connections beyond traditional public health datasets and can position health departments and communities to understand a wide range of social and economic effects during a public health crisis through the inception and recovery phase.