Effects of the COVID-19 crisis on survey fieldwork: Experience and lessons from two major supplements to the U.S. Panel Study of Income Dynamics

Abstract

Two major supplements to the Panel Study of Income Dynamics (PSID) were in the field during the COVID-19 outbreak in the United States: the 2019 waves of the PSID Child Development Supplement (CDS-19) and the PSID Transition into Adulthood Supplement (TAS-19). Both CDS-19 and TAS-19 abruptly terminated all face-to-face fieldwork and, for TAS-19, shifted interviewers from working in a centralized call center to working from their homes. Overall, COVID-19 had a net negative effect on response rates in CDS-19 and terminated all home visits that represented an important study component. For TAS-19, the overall effect of Covid-19 was uncertain, but negative. The costs were high of adapting to COVID-19 and providing paid time-off benefits to staff affected by the pandemic. Longitudinal surveys, such as CDS, TAS, and PSID, that span the pandemic will provide valuable information on its life course and intergenerational consequences, making ongoing data collection of vital importance.

1. Introduction

We describe how two major supplements to the Panel Study of Income Dynamics (PSID) that were both in the field at the time of the COVID-19 outbreak in the United States adapted to the pandemic and the mandated lockdown. The two supplements are the 2019 waves of the PSID Child Development Supplement (CDS-19) and the PSID Transition into Adulthood Supplement (TAS-19). Both surveys are still in the field, as of May 2020, with fieldwork scheduled to continue through to June and July, respectively, for CDS-19 and TAS-19. We focus on the response and adaptation of fieldwork operations for both studies following the emergence of COVID-19 and the imposition of lockdowns, and our plans for completing data collection efforts.

The major effects of the pandemic and lockdown (henceforth, COVID-19) for both CDS-19 and TAS-19 were to abruptly terminate all face-to-face fieldwork effort and, for TAS-19, to shift interviewers from working in a centralized call center to working from home. We responded to the former effect by developing a plan to collect face-to-face content remotely and to drop content that could not be adapted to this mode. The shift of interviewers to working from home went smoothly, due to coordinated efforts from a number of different work groups. Finally, an opportunity emerged in CDS-19 to collect new measures about the effects of COVID-19 from a subsample of cases for whom in-person visits were pending at the time of the fieldwork transition. For both studies, a number of challenges remain for completing fieldwork, processing the data, and having data users capitalize on the valuable information that was collected.

In the next section, we provide a brief description of PSID, CDS, and TAS. We then describe the fieldwork plans, staffing model, and timelines prior to COVID-19. This is followed by a description of the impact of COVID-19 on fieldwork protocols and operations for the two PSID supplements. Next, we describe the adaptations that were implemented and the plans to complete fieldwork for CDS-19 and TAS-19. We end with a summary and discuss some implications.

2. Overview of PSID, CDSs, and TAS

PSID began in 1968 and is the world’s longest-running household panel survey. Since its inception, it has been based at the University of Michigan (U-M). PSID was established to study income dynamics with a U.S. national sample of approximately 5,000 families. The sample has grown over time as children born to original sample members reached adulthood, formed their own independent households, and were invited to join the panel as primary respondents (Johnson, McGonagle, Freedman, & Sastry, 2018). A new immigrant refresher sample comprising approximately 500 families was added to PSID in 1997 and again in 2017. The panel now includes over 10,000 families and the topics covered in the current biennial interview have expanded to also include factors that shape and are determined by income dynamics—such as, education, health, and time use.

The availability in PSID of detailed panel data on this broad set of factors led researchers to ask how they shaped children’s health, development, and well-being. In 1997, PSID launched the Child Development Supplement to collect data to answer these questions (McGonagle & Sastry, 2015). CDS has been repeated at approximately five-year intervals. It initially followed a cohort of children, but since 2014 has collected data on all children in PSID families.

As individuals enter early adulthood, they make key decisions about education, employment, family formation, and residential independence. In 2005, PSID launched the Transition into Adulthood Supplement to collect data on these decisions, in order to study both their childhood antecedents (with data from CDS) and the life course consequences (with data from PSID). TAS thus plays an important role in bridging the content of CDS and PSID and in tracking sample members over their early life course. It has been conducted biennially, initially on the original cohort of CDS children but since 2017 on all young adults aged 18–28 years in PSID families.

CDS, TAS, and PSID are released as public use data and have been used in over 6,400 peer-reviewed publications. The lead sponsor for PSID is the U.S. National Science Foundation, while CDS and TAS are funded by the U.S. National Institutes of Health.

3. Fieldwork plans for CDS and TAS

Eligible respondents for CDS-19 and TAS-19 were identified in PSID-19, which began fieldwork in Spring 2019 and ran through the end of the calendar year. Fieldwork for CDS-19 and TAS-19 was launched in Fall 2019, with additional cases released through the spring of 2020. Some PSID families were eligible for both CDS-19 and TAS-19. All fieldwork was conducted by the Survey Research Operations (SRO) group at U-M.

CDS-19 was designed as a multi-component study combining telephone and in-person data collection. Telephone components included a 10-minute screening interview to identify all eligible children and their primary caregiver (PCG, usually a child’s parent); a 75-minute survey with the PCG about the household and each eligible child; and a 40-minute survey with adolescents aged 12–17 years. In-person data collection in participants’ homes (hereafter, home visits) included anthropometric measurements; two 24-hour time diaries for each child; interviews with children aged 8–11 years; consent forms to obtain children’s school records and children’s and PCGs’ birth records; saliva samples for subsequent genetic analysis from children aged 5–17 years, PCGs, and other adults in the household; and achievement tests for children aged 3–17 years (verbal and quantitative reasoning) and PCGs (passage comprehension). In some cases, interviews with PCGs and, especially, adolescents were completed during the home visit if these had not already been completed by telephone.

A decentralized, nationally-distributed team of 83 CDS interviewers worked from home to administer telephone-based study components and traveled to respondents’ homes. Interviewers focused initially on completing telephone interviews and on conducting home visits in their local areas. Trips further afield, by car or air, were to begin only after most telephone interviews were completed. Families in small clusters and in locations far from an interviewer were subsampled.

CDS-19 fieldwork was scheduled for 1 October 2019 to 31 May 2020. The initial sample included 1,584 households whose PSID interview was already complete. The balance of the sample was added in three batches between late October and early February. The total final sample comprised 3,526 households, 3,088 of which were eligible for home visits.

The study design and staffing model for TAS differs from CDS. TAS interviews two types of respondents: those who were PSID-19 main respondents and those who had not yet established independent households. The former group comprises approximately 45% of the sample and receives a 42-minute interview that skips content on certain topics, such as employment and health, that was collected for these respondents in their PSID-19 interviews. The latter group comprises the remaining 55% of the sample and receives the full TAS interview, which is approximately 66 minutes in duration.

TAS-19 uses a mixed-mode design that randomly assigned 80% of the sample to a web-first mode and the remaining 20% to telephone with no web option. In-person interviews were planned for a small subset of cases that lacked updated telephone numbers and email addresses. Web cases that had not completed an on-line interview by the eighth week of fieldwork began receiving telephone calls, in addition to continuing email and text messages reminders they had received since the beginning of fieldwork. Calls and messages encouraged respondents to complete the survey online, while also offering the option of completing the interview by telephone. These messages were sent several times a week.

Nineteen interviewers employed by the Survey Services Lab (SSL), a telephone call center run by SRO, were trained for TAS-19. All interviewers worked from carrels in the lab. The call center assigned cases to interviewers on a flow basis, using an automatic scheduler that selected cases for calling based on contact history and the status of all remaining cases.

TAS has a total sample of 3,005 cases. Fieldwork for TAS-19 began on 13 November 2019 with an initial release of 2,600 cases. TAS-19 data collection was scheduled for approximately six months, through the late spring or early summer of 2020. Fieldwork protocols were also designed to manage approximately 1,000 cases that were eligible for both CDS-19 and TAS-19, with TAS receiving initial prioritization, and some cases flagged for simultaneous work.

4. Impact of COVID-19 on fieldwork for CDS and TAS

Project investigators were following the COVID-19 outbreak in early 2020. By early March 2020, SRO staff overseeing the home visits for CDS-19 began to have concerns about areas in the U.S. that were first affected by COVID-19. Home visits were suspended on March 6 in the Seattle area in Washington State and a few days later in other areas with early outbreaks, including the San Francisco Bay Area in California and the New York City suburbs. All longdistance and overnight travel was also canceled, and only previously scheduled local home visits proceeded. The last CDS-19 home visit occurred on March 13, one day prior to the suspension of all face-to-face human subjects research by U-M’s Institutional Review Board (IRB). At the time of the IRB suspension, home visits had been completed with 902 families—about 34% of cases that were eligible for home visits and had completed all other study components. These cases were clustered in larger cities where interviewers were locally based and do not represent a random subsample of all CDS-19 cases.

The pace of change demanded by COVID-19 posed an unprecedented challenge to fieldwork, and CDS-19 benefitted from exceptionally effective responses from PSID and SRO staff, field interviewers and their supervisors, and the IRB. Because of their acuity and foresight, telephone-based data collection continued almost without interruption. One significant cost to the project has come through paid time-off benefits for regular and contingent employees who were infected with COVID-19, had to care for an infected family member, or were unable to work due to other commitments related to COVID-19 (such as caring for a child due to school closure). U-M and the federal government, through the Emergency Paid Sick Leave Act (EPSLA), obligated CDS-19 to pay for these benefits. To date, these costs have totaled several hundred thousand dollars.

The impact of COVID-19 on fieldwork protocols and operations for TAS-19 was significant, but, because of its lower use of face-to-face protocols, less consequential than for CDS-19. On 13 March 2020, a general directive by U-M required all non-essential staff to work remotely, with only essential staff permitted to come into the office on a limited basis. All call center interviewers and supervisory staff were thus shifted to conducting remote work from their homes. Some advance warning of the impending directive facilitated the equipping and training of interviewers for remote work. Interviewer training provided instruction for remotely accessing the SSL workstations, which avoided transmitting respondents’ personally identifying information to computers in the interviewers’ homes and allowed the continued use of the call center’s automatic scheduler to assign and prioritize cases.

Although TAS-19 did not have extensive plans to conduct in-person visits to track or interview respondents, the IRB’s ending of all face-to-face interactions with respondents led to the project canceling all of TAS-19’s existing plans for in-person visits. This almost certainly had a negative effect on TAS-19 response rates, particularly for respondents without high-quality contact information other than a name and address.

5. Fieldwork adaptation in response to COVID-19 for CDS and TAS

After CDS-19 in-person data collection was suspended, fieldwork focused on telephone interview components and follow-up efforts. Screening interviews ended on April 19 (85% response rate), interviews with PCGs ended on May 10 (86% response rate), and interviews with adolescents are scheduled to end on May 31 (>60% expected response rate).

In response to the COVID-19 suspension of home visits, CDS-19 initially planned a follow-up home visit effort in Fall 2020 to collect information from the remaining 62% of eligible families. However, this plan faced significant threats from the likelihood of a second wave of COVID-19 cases leading to new lockdowns and expected reluctance or refusal from families to allow interviewers into their homes. Because of the high cost of planning, staffing, and launching an in-person component, and the limited flexibility once underway, the study team decided to shift the associated data collection efforts to a telephone and mail-out/mail-back protocol.

Planning is now underway for a follow-up effort beginning in September 2020 that will remotely collect all components from the home visit with the exception of achievement tests, which could not easily be adapted to remote administration. The remaining study components can feasibly be collected by telephone or through a mail-out/mail-back protocol. This approach will allow us to collect data from all eligible respondents, but we anticipate that response rates will be lower and may be correlated with a variety of factors. Fortunately, data from PSID-19 and CDS-19 telephone components provide substantial information about all families in the sample, which will allow us to construct an appropriate set of non-response weights. Another potential limitation of this protocol is that data quality may be affected by measurement and recording errors that arise when study participants are asked to complete tasks that would otherwise be managed by trained interviewers.

We will use the Fall 2020 follow-up effort as an opportunity to administer a short module that collects information on the effects of COVID-19 on families and children. The module will include questions about diagnosis of and testing for COVID-19 among family members as well as economic, behavioral, and social impacts of COVID-19 on children and their caregivers.

Ongoing TAS-19 operations were affected by the response to COVID-19 in multiple ways, with effects on staff productivity, administrative issues related to data collection and operations, and fieldwork outcomes. Although technical aspects of the transition to remote work went smoothly, the abrupt transition led to an immediate reduction in interviewer hours because staff needed additional training and time to learn the new procedures. Supervisory staff also required time to adjust to remote oversight of interviewer staff. At the same time, there were reductions in overall productivity as staff coped with closures of schools and child-care facilities, as well as increasing caregiving demands for family members and children. Minor budgetary losses were incurred by staff use of paid time-off related to COVID-19 that was mandated by U-M and the federal government (but paid for directly by data collection projects).

The shutdown reduced the availability of administrative support, which, in turn, led to delays in new and ongoing data collection procedures. For example, there were delays in printing, preparing, and sending respondent mailings, processing of respondent incentive payments, and obtaining approvals for using external vendors to conduct respondent tracing. The IRB required TAS-19 (and all other active projects) to submit descriptions of new data collection protocols for remote work to ensure appropriate protection of human subjects, including the handling of confidential data. There were minor disruptions to data collection and project management as staff developed new protocols and submitted them to the IRB for review and approval.

It is challenging to estimate empirically the effects of COVID-19 on TAS-19 fieldwork outcomes due to the shift to remote work and ensuing reduction in staff hours and the delays in implementing various protocols. Not only were the professional and personal lives of project staff affected by the ongoing pandemic, but so too were the lives of respondents. On the other hand, due to state and local “stay-at-home” orders, respondents may have been more likely to be at home and welcoming of the incentive payments. TAS-19 developed new materials to acknowledge the disruption in respondents’ lives by COVID-19 through amending a mailing encouraging participation that was already in preparation. A note was added to the envelope exterior indicating that the contents contained “An important update from the University of Michigan.” The inside message highlighted that “in response to the current situation, interviewers are still able to work safely” and were eager to talk with respondents.

On balance, and despite the disruption due to COVID-19, TAS-19 is on-track to achieving its fieldwork goals. Through early May 2020, TAS-19 has achieved a response rate of 73%. Sufficient time remains in the field period to approach the target of an 89% response rate.

6. Summary and conclusions

Overall, we assess that COVID-19 had a net negative effect on response rates in CDS-19. The lockdown did mean that respondents were more likely to be home and incentive payments were especially welcome. At the same time, parents faced challenges in working from home, caring for children, supervising children’s activities, and home schooling that precluded them from participating in the survey. The net result was certainly some loss of scientific data, specifically related to the modules collected in-home that could not be continued. The opportunity to collect new information on COVID-19 impacts in Fall 2020 is a potential ameliorating benefit.

For TAS-19, the overall effect of COVID-19 was uncertain. Although we cannot easily estimate a causal effect on response rates, the net effects were likely modest but certainly negative. We are on-track to completing fieldwork generally as planned and expect to achieve a response rate close to our goal. The lack of in-person visits means that we are likely to have higher non-response among young adults without telephones or internet access. And the large disruption in shifting to off-site work led to lower productivity, higher costs, and lost opportunity to develop and implement innovations to address fieldwork challenges.

The two survey components described here—and PSID more generally—were able to adapt quickly to the COVID-19 situation by switching to a telephone-only mode and entirely eschewing in-person visits. This was possible because of our initial design (including our major web component on TAS-19). However, the situation did preclude the collection of some significant in-person components that could not be collected remotely.

A consequence of COVID-19 is that future work on PSID and its CDS and TAS supplements—as well as other panel surveys—will focus on new and effective ways to collect data remotely while addressing respondent concerns and achieving interactions with respondents in ways that are typically only possible with in-person visits. For instance, we foresee greater use of video-based calls to recruit respondents, address respondent concerns, or coordinate collection of more complicated components such as anthropometric measurements, collection of saliva samples, or conducting skills assessments. We are also exploring ways to use the web more effectively, not just to conduct interviews, but also to obtain signed consent forms that use digital signatures, complete time diaries for children, and collect sensitive data from adolescents.

COVID-19 is, hopefully, a once-in-a-lifetime event experienced by PSID and other surveys. Although disasters and disruptions routinely occur in on-going surveys, they tend to be short-term or geographically localized events, such as hurricanes, floods, earthquakes, and other natural disasters. COVID-19 is of course remarkable in its national and global scope and its open-ended duration. Longitudinal data collected by CDS, TAS, and PSID from both before and after the pandemic will, we hope, shed important light on its life course and intergenerational effects.