Survey Breakoffs in a Computer-Assisted Telephone Interview

Abstract

Nearly 23% of all telephone interviews in the most recently completed wave of the Panel Study of Income Dynamics break off at least once, requiring multiple sessions to complete the interview. Given this high rate, a study was undertaken to better understand the causes and consequences of temporary breakoffs in a computer-assisted telephone interview setting. The majority of studies examining breakoffs have been conducted in the context of self-administered web surveys. The present study uses new paradata collected on telephone interview breakoffs to describe their prevalence, associated field effort, the instrument sections and questions on which they occur, their source – whether respondent-initiated, interviewer-initiated, or related to telephone problems – and associations with respondent and interviewer characteristics. The results provide information about the survey response process and suggest a set of recommendations for instrument design and interviewer training, as well as additional paradata that should be collected to provide more insight into the breakoff phenomenon.

1 Introduction

Survey breakoffs occur when an interview with a respondent starts but then stops before completion. Empirical examinations of breakoffs have generally been conducted in the context of self-administered web surveys (e.g., Ahsan and Broach 2005; Peytchev 2009, 2011; Sakshaug and Crawford 2009), finding median breakoff rates between 16% and 34% (Musch and Reips 2000; Lozar-Manfreda and Vehovar 2002). The availability of newly collected paradata on breakoffs in the Panel Study of Income Dynamics (PSID) shows that the prevalence is also relatively high in a computer-assisted telephone interview (CATI) setting, with nearly 23% of all completed interviews in the most recent wave breaking off at least once. While much can be learned about this phenomenon from the web experience, it is likely that the causes and consequences of breakoffs differ in a CATI setting. Web breakoffs often lead to unit nonresponse (Peytchev 2009) and are sometimes referred to as ‘abandonments’, or ‘terminations’, while telephone interviews that breakoff or suspend may eventually be completed through interviewer efforts (Groves 1990; Groves and Couper 1996), although substantial field costs may be incurred (Keeter et al. 2006). Moreover, with the exception of technical failure, web survey breakoffs are the result of a respondent-driven action, while CATI breakoffs may have multiple causes, involving the behavior of interviewers as well as respondents. Prior wave participation of respondents and features of the survey instrument may also affect survey breakoff. Thus, in a CATI setting, the causes and consequences of interview breakoffs may be multiple and complex.

Given the high level of temporary breakoffs observed in the PSID, an observational study was undertaken to better understand their causes and consequences, including their prevalence, associated field effort, the instrument sections and questions on which they occur, and their associations with characteristics of respondents, interviewers, and features of the survey instrument. Newly available paradata starting with the 2011 wave also provided information about the source of the breakoff – whether respondent-initiated, interviewer-initiated, or related to telephone problems – permitting a detailed exploration of the causes and consequences of these different types of breakoffs. The results provide a broad context for understanding temporary survey interview breakoffs that can be used to suggest survey instrument modifications and identify training needs of interviewers. To the extent that such improvements reduce respondent and interviewer burden, breakoffs may become less prevalent in future waves, allowing the data to be collected more efficiently. The paper also provides suggestions about how breakoffs as a survey response behavior may fit within models of survey participation, and proposes avenues for future research.

2 Background

Existing theoretical frameworks of survey participation (Groves and Couper 1998; Lepkowski and Couper 2002; Groves et al. 2000) and the handful of studies that have specifically examined instrument design features and respondent characteristics associated with breakoffs (Ahsan and Broach 2005; Keeter et al. 2006; Klein et al. 2011; Peytchev 2009, 2011; Stussman et al. 2004) informed the overall design of this study and the derivation of hypotheses about the underlying associations of these factors with breakoff propensity. As the theoretical framework posited by Groves and Couper (1998) describes, influences on survey participation are multi-faceted and include features of the survey instrument, interviewer characteristics, attributes of the respondent, and the environment in which the interview is taking place. Moreover, these influences include reciprocal associations between each of these factors, such as the effects of respondent and interviewer characteristics on perceptions of features of the survey instrument. In combination with findings from the research literature, this framework is used to define the possible factors affecting the propensity for breakoff in the current study, and help guide expectations for the direction of the effects.

First, several features of the survey instrument were expected to be associated with breakoff propensity. Drawing on research based on web survey breakoffs, sections of the instrument that are cognitively burdensome such as those with questions requiring mental calculations and lengthy recall periods (Peytchev 2009, 2011) were expected to be associated with increased breakoff rates. Questions involving statements intended to introduce a new content area (i.e., “introductory questions”) were also expected to have high breakoff rates based on research documenting such effects in telephone surveys (Groves and Kahn 1979), telephone data entry surveys (Tourangeau et al. 2002), and web surveys (Peytchev 2009). Finally, the only study of which we are aware that has examined the relationship between interview length and breakoff found that the cumulative number of questions asked in a web survey had no effect on breakoff (Peytchev 2009). Interview length has been found to predict sample retention in a face-to-face panel study potentially due to its relationship to respondent interest rather than burden (Branden et al. 1995). However, in the context of a telephone interview, high interview length was expected to increase breakoff propensity as telephone batteries depleted, and respondents became fatigued or had other commitments.

Second, drawing on the theoretical framework of survey participation proposed by Groves and Couper (1998) highlighting the primary importance of interviewer behavior in gaining the cooperation of respondents, as well as substantial research evidence (e.g., Laurie et al. 1999; Groves and McGonagle 2001), we expected that breakoffs which were initiated by interviewers would be relatively infrequent, given an interviewer’s motivation to keep the respondent on the telephone and move the interview to completion. Given the complexity of many of the sections of the PSID computer-assisted telephone instrument, including a lengthy and nontraditional event history calendar (see Beaule et al. 2009), we further expected that while rare, breakoffs would be higher among interviewers with less interviewing job experience.

Third, the increasing reliance on cell phones as primary telephones in the U.S. (Pew Internet Survey 2012), combined with a lengthy interview such as PSID that lasts more than an hour on average, led to the expectation that telephone problems such as battery depletion, as well as switching from a cell phone to a landline in order to avoid battery failure, would generally be responsible for a large share of breakoffs and would increase in likelihood over the course of the interview.

Fourth, several socio-demographic characteristics of respondents were also expected to be associated with breaking off. For example, some respondent characteristics that may be indicative of a complex life, such as having high income and many assets, multiple jobs, and many family members, and/or changes in family structure, were expected to be associated with breakoff potentially through an effect on interview length, with the number of questions asked during the interview increasing as a function of these life characteristics, or simply because such respondents have greater scarcity of time to participate in a lengthy interview. The relationship between breakoff and age was not immediately apparent, with research finding increasing age in the absence of health problems generally related to higher cooperation (e.g., Groves et al. 2000) but also to higher rates of item-missing data, potentially attributable to declining information-processing ability (Knauper et al. 1997). Breakoff rates were expected to be higher among those with relatively low levels of education, based on prior research demonstrating this association especially for complex interview content (Keeter et al. 2006; Peytchev 2009). In the context of a panel study, an individual’s prior wave survey experience is known to influence each successive survey request (e.g., Lepkowski and Couper 2002; Peytchev 2011; Uhrig 2008). Influences of prior wave field effort that may signal respondent reluctance to participate in the interview were expected to increase breakoff, including prior wave breakoff and nonresponse, expressing resistance, needing tracking, high interview length, and high number of calls to final disposition.

3 Methods

3.1. The data

This study draws on data collected during Waves 36 (in 2009) and 37 (in 2011) of the Panel Study of Income Dynamics (PSID). The PSID is a longitudinal study of a nationally representative sample of U.S. families that began in 1968 and collects a variety of data on economic, health, and social behavior (see McGonagle et al. 2012 for more information). The original 1968 PSID sample was drawn from a nationally representative sample of roughly 3,000 families designed by the Survey Research Center at the University of Michigan (the “SRC sample”) and an over-sample of roughly 2,000 low-income families from the Survey of Economic Opportunity (the “SEO sample”). In 1997, 511 families who had immigrated to the U.S. after 1968 were added to enhance the national representativeness of the sample.

The survey interview is conducted with one respondent within the family, typically the household head or spouse, who reports primarily factual, non-attitudinal information about the family as a whole and on individual family members. PSID has conducted interviews with its families annually 1968–1997, and biennially since 1999 through the most recent wave in 2013. Data collection occurs in oddnumbered years between March and December. Since 2003, the mode of data collection for approximately 97.5% of the sample is via computer-assisted telephone interview (CATI) using Blaise software, along with SurveyTrak, a sample management system developed by the University of Michigan, with 2.5% of the interviews conducted in person with individuals who are unable to be interviewed via telephone. During 2011, interviews were completed with 8,941 families by 120 field interviewers employed by the University of Michigan. The field interviewers reside all across the U.S. and typically conduct the telephone interviews from their homes.

The analyses in this paper are based on 8,311 families who were eligible to be followed as of Wave 37 in 2011 and provided a completed interview. This reflects an intentional focus in this paper on temporary breakoffs that occur within completed interviews. In Wave 37 of PSID, less than 5% of interview breakoffs resulted in nonresponse. The majority of the 8,311 families also provided completed interviews during Wave 36 in 2009, reflecting the very low rate of between-wave nonresponse in the PSID.

3.2. Measures

Measurement of breakoff characteristics

Paradata on interview breakoff collected by the SurveyTrak sample management system during production interviewing in 2011 is the key outcome of this study. The data included: 1) whether a breakoff occurred, 2) the specific variable in the instrument on which the breakoff occurred, which also permitted identification of the questionnaire section of the breakoff, and 3) the source of the breakoff, as recorded by the field interviewer. Interviewers used a predefined set of codes to record the source of the breakoff based on whether it was initiated by the respondent, interviewer, or occurred due to a telephone problem, which included disconnections, batteries and prepaid minutes elapsing, and telephone switches such as from cell phone to landline and vice versa. Interviewers also provided a text-based description for approximately 25% of all breakoffs, which, as noted by Stussman et al. (2004), provides insightful information about the reasons underlying the breakoff. For example, reasons for respondent breakoffs included: Respondent was tired/hungry/needed to ask spouse for information/needed to go to work/needed to care for child/ ...etc. Reasons given for suspending the interview by interviewers themselves fell into three main categories: difficulty or confusion navigating particular sections of the instrument, experiencing a technical problem with the instrument software or laptop (i.e., instrument not advancing; laptop “freezing”), and needing clarification on a procedural issue from a supervisor before continuing with the interview, typically regarding the completion of the household roster at the start of the interview.

Since about two-thirds of interviews with a breakoff had exactly one and the paradata are unambiguous in identification of the source and location of the first breakoff, analyses of the source of the breakoff and its location in the questionnaire are based on the first breakoff.

Measurement of introductory questions

The analyses described in Section 4.5 examine rates of breakoffs on introductory questions. Introductory questions are those that are designed with the intent to provide a smooth conversational transition between different content areas in an interview. A review of the 1,381 questions that were included in the 2011 PSID questionnaire identified 36 questions (2.6% of all questions) as being ‘introductory’ in nature through their foreshadowing of the upcoming question series, using descriptions such as: “Next are questions about…” and, “Now I will ask you about…”. Most of the sections in the instrument had at least one such introductory question, and there were often multiple introductory questions embedded within sections that were particularly lengthy, as well as in sections that had multiple but related content domains.

Measures used to examine predictors of having a breakoff interview

Measures used in multivariate models examining predictors of interview breakoff (Section 4.6) included: socio-demographic characteristics of respondents and families, prior wave field effort, and current wave cell phone use, interview length, and interviewer experience.

Characteristics of respondents and families used in multivariate models included: total family income, education, age, number of jobs held in the past two years by the household head, whether the family had experienced a change in its composition since the prior wave, family size, and time invariant indicators for sample type. Family income. A series of sensitivity analyses examined the cumulative effect of income on rate of breakoffs; each additional quartile was associated with a linear increase in breakoff rate. Based on these results, an indicator variable was created for whether the total family income was equal to or above the fourth quartile in total family income (‘yes=1’/ ‘no=0’). Education of household head. A dummy variable coded whether the household head had completed some college or more (‘yes=1’) or high school or less (‘no=0’). About half of all household heads had completed some education beyond high school. Number of jobs held. A dummy variable coded whether the household head held three or more jobs in the past two years (‘yes=1’, ‘no=0’), with about 4% of household heads coded ‘yes=1’. Age of household head. A dummy variable coded whether the age of the household head was under age 46 (‘yes=1’) or age 46 and older (‘no=0’), with approximately 55% of household heads under age 46. Family composition change. A dummy variable coded whether the family experienced a change in composition (i.e., new member(s), member(s) died or moved out, a change in household head) since the prior wave (‘yes=1’, /’no=0’). Nearly 37% of all families had a change. Family size. A dummy variable coded whether there were more than two individuals in a family unit (‘yes=1’, ‘no=0’), with about 44% of all families coded ‘yes’ on this variable. Sample type. Two separate dummy variables that each identified types of sample members were included in the models. The first dummy variable coded families who were part of the original low-income Survey of Economic Opportunity (SEO) oversample as ‘yes=1’ and coded families who were part of the original Survey Research Center (SRC) national probability sample as ‘no=0’. A second dummy variable coded families who came from the 1997/1999 post-1968 immigrant refresher sample as ‘yes=1’ and those who did not come from this sample as ‘no=0’. Approximately 62 percent of the families are from the SRC sample, 30 percent from the low-income SEO over-sample, and 7 percent from the immigrant sample.

A second set of variables were created to assess the effect of prior wave field effort on the occurrence of breakoffs. Prior wave breakoff. A dummy variable coded whether a breakoff occurred in 2009 (‘yes=1’, / ‘no=0’). The rate of interviews with at least one breakoff was 13.7% in 2009. Prior wave non-response. A dummy variable coded whether the case was nonresponse (‘yes=1’, / ‘no=0’). Among the 8,311 families in the analysis who all provided interviews in 2011, 1.5% were nonresponse in 2009. Interim refusal is a dummy variable that provides information about whether the case was one of the 3.5% in 2009 who ever gave a refusal that was subsequently converted to a final interview (‘yes=1’, / ‘no=0’). Prior wave number of calls. A dummy variable was included for cases that received more calls to finalize than the median of 6.0. Tracking required captured information about whether the sample case needed specialized interviewer effort during the field period to be located, whether due to the telephone number of record not being answered or out of order, or the respondent moved and no forwarding contact information was available. The variable is a dummy variable that allows the calculation of a tracking rate in the sample (‘yes=1’, / ‘no=0’). Twenty-two percent of the families required tracking in 2009.

A dummy variable was included for whether a cell phone was used on any call during the 2011 field period (‘yes=1’, / ‘0=no’). More than 70% of all interviews in 2011 included the use of a cell phone on at least one call.

A series of three dummy variables, each coded as ‘yes=1’, and ‘no= 0’, captured the length of the interview in 2011 using the following cut-points based on a sensitivity analysis: equal to or below the 50^th percentile (21–84 minutes), between the 50^th and 75^th percentiles (85–104 minutes), between the 75^th and 90^th percentiles (105–127 minutes) and above the 90^th percentile (128–487 minutes).

The final variable was a dummy variable for whether the interviewer was one of the 36% during 2011 that was newly hired (‘yes=1’) or had prior interviewing experience (‘no=0’).

3.3 The PSID computer-assisted telephone instrument

This section provides a brief overview of the PSID 2011 instrument as context for the analyses to follow. The instrument was comprised of 12 distinct topical sections with an overall average length of 93.7 minutes. Table 1 shows the average length of each section in order of its administration and its overall percentage of the total instrument. With the exception of sections asking about new marriages, births, and family members, all respondents were administered every section, with check points routing respondents to various question paths depending on their answers to prior questions. Thus, the sections vary in both average number of minutes to complete and number of questions.

Table 1.

Section lengths of the 2011 PSID instrument

Section	Section length (minutes)	Total instrument (%)
Household Roster	4.3	4.6
Housing	6.6	7.0
Employment & Residence calendar	24.3	26.0
Expenditures	10.1	10.8
Income last year	8.8	9.4
Income two years ago	2.0	2.1
Wealth	6.4	6.9
Pensions	4.1	4.4
Health	18.4	19.6
Marriages and Births	1.2	1.3
New family information	2.5	2.7
Philanthropy	5.0	5.3
Total	93.7	100.0

The interview begins with an update of the roster of all individuals living in the household using a grid of information preloaded from the prior wave. Interviewers are trained that accurate completion of this section is of primary importance as it determines the correct selection of the respondent and the passing forward of text throughout the instrument that fills subsequent questions about the respondent and other family members (e.g., to include first names and reflect gender) and, in married couples, adds questions about the spouse. The first section collects basic information on housing including dwelling characteristics, utilities, rent and mortgages. The following section collects information on employment and residence. This section is the lengthiest of the instrument and interviewers consistently report that the questions are challenging for respondents as they require the dating of events for periods that may exceed 24 months, as well as mental calculations. Many of these questions are asked in the framework of an event history calendar which captures timing information about residential and job changes over the past two calendar years (for more detail see Beaule et al. 2009). Characteristics of this section make it unique and complex for interviewers as well. While interviewers are trained to administer the majority of the instrument reading each question exactly as written and in a defined order, the calendar permits the use of a conversational interviewing style, and allows interviewers to navigate freely between different questionnaire screens. Moreover, data entry within the calendar structure of month and day information using a timeline is fundamentally different from the data entry that typifies the majority of the instrument. A series of five sections on economic behavior and finances follows the calendar, including complex questions about expenditures, detailed components of income, wealth and active savings, and employer pensions. The interview then collects information on health in a lengthy section that asks about health status, behaviors, conditions, insurance coverage, and expenditures. The final substantive section collects information on philanthropic behavior.

4 Results

Five sets of results are presented below. Implications of the findings are addressed in the Discussion section. First, the prevalence of interviews completed in Wave 37 of the PSID that included at least one breakoff and the prevalence of breakoff types – whether respondent-based, interviewer-based, or phone-based – is described in Section 4.1. This section also provides information on the prevalence of breakoffs and breakoff types among the field interviewers who participated in Wave 37. Second, Section 4.2 describes production characteristics during data collection in 2011 by whether an interview had a breakoff, and examines patterns of field effort by breakoff type. Third, information on the section of the survey instrument in which all first breakoffs occur and by breakoff type is presented in Section 4.3. Fourth, results examining the prevalence of breakoffs on ‘introductory’ questions compared to non-introductory questions are presented in Section 4.4. Finally, Section 4.5 presents results from multivariate models examining how respondent and interviewer characteristics and prior wave field effort predict the occurrence of at least one breakoff. Multinomial logistic regression analysis is used to model the type of breakoff as a nominal variable consisting of three categories: respondent-based, interviewer-based, or phone-based.

4.1 Breakoff prevalence and type

The prevalence of interviews with at least one breakoff during the 2011 field period was 22.7%. The breakoff rate in 2009, the only other wave for which paradata on breakoffs are available, was substantially lower at 13.7%. Thus, of the 8,311 interviews completed in 2011, 1,889 interviews had at least one breakoff, requiring more than one session to complete.

While the instance of first breakoff can occur exactly once for each respondent, field interviewers may administer multiple interviews that have breakoffs. The number of completed interviews conducted by the 120 interviewers that participated in the 2011 PSID field effort ranged from 1 to 190 with an average of 69 completed interviews per interviewer. All but 5 interviewers had at least one interview with a breakoff. Interviewers conducted an average of 16 interviews with at least one breakoff, with a median of 14, and a maximum of 47. As would be expected, there was a strong positive association between the number of interviews conducted and breakoff rate (r=.80, p<.0001). These results indicate that interviews with breakoffs are not concentrated in a small group of interviewers, but are instead widely experienced across the field staff.

Table 2 shows that about 60% of all first breakoffs were initiated by respondents, 21% were interviewer-initiated, and about 19% were due to a telephone problem. Among the 115 interviewers who had at least one interview with a breakoff, most had breakoffs of each type: nearly all had at least one respondent-based breakoff (97%), and most initiated at least one themselves (84%) and had at least one due to telephone problems (81%).

Table 2.

Source of first breakoff

	n	%
Respondent	1132	59.9
Interviewer	404	21.4
Telephone Problem	353	18.7
Total	1,889	100.0

4.2 Production characteristics associated with breakoffs

Field effort

Table 3 presents production characteristics for completed interviews by whether or not the interview had a breakoff. The results show that completed interviews with at least one breakoff are characterized by substantially greater field effort than interviews obtained in one session. In fact, these differences are quite large: interviews with a breakoff are substantially longer on average by more than 20% or by 18 minutes (102.5 vs. 84.5 minutes), require nearly double the average number of calls to finalize (20.0 vs. 11.1), and are significantly more likely to require specialized tracking and refusal aversion efforts (25.8% vs. 19.5%, and 5.7% vs. 2.6%, respectively). Finally, while cell phone use in PSID is high generally, interviews with breakoffs are significantly more likely to involve the use of a cell phone (79.4% vs. 67.7%), which may itself be a key reason for the breakoff as shown below.

Table 3.

Production characteristics in 2011 by breakoff status

	At least 1 breakoff?
2011 Production characteristics	Yes	No	Difference
Average interview length (Min.)	102.5	84.5	18.0
Average number of calls	20.0	11.1	8.9
% with refusal on any call	5.7	2.6	3.1
% requiring tracking	25.8	19.5	6.3
% using cell phone on any call	79.4	67.7	11.7
n	1,874	6,398

^*All differences are statistically significant at p<.0001.

Timing of effort relative to breakoff

An examination of the number of calls before and after the breakoff, and the timing of the call on which tracking and refusal aversion efforts were applied relative to the timing of the call of first breakoff provides information about whether field effort is a cause or consequence of breaking off. That is, are the higher number of calls and tracking and refusal aversion rates observed for breakoff cases a consequence of effort due to recontacting the case after the breakoff has occurred, or, are the cases that breakoff those that tend to require high effort at the outset? The results indicate that these cases require high effort at the outset, with most of the field effort concentrated in the time period preceding breakoff, with about 70% of all the calls occurring before the first breakoff. Tracking also typically precedes breakoff, occurring on average within the first five calls as interviewers learn that they have a non-working or out-of-date telephone number, as may happen when a respondent has moved with no forwarding contact information. Although most of the field effort occurs prior to first breakoff, these cases do also show substantial effort following the breakoff, especially those requiring tracking or refusal aversion. For example, breakoff cases in tracking have 10 calls on average after the first breakoff before being finalized. Similarly, those that have received an interim refusal indicator have an average of nearly 45 calls, about half of which occur after first breakoff.

Field effort and source of breakoff

Table 4 displays production characteristics by source of first breakoff. The majority of significant differences are between respondent-based breakoffs and telephone breakoffs. There are no differences in interview length between any of the breakoff types. Respondent-based breakoff interviews require significantly more calls to finalize compared to those breaking off due to telephone problems, and are more likely to be associated with at least one interim refusal compared to breakoffs for other reasons, although significantly so only when compared to interviewer-based breakoff interviews. Interviews with a first breakoff due to telephone problems require significantly more tracking than those that breakoff due to respondent reasons. Finally, and not surprisingly, the highest rate of cell phone use is by interviews with a first breakoff due to telephone problems, at about 86%, compared to those due to respondent reasons at 79% and interviewer reasons at about 76%.

Table 4.

Production characteristics in 2011 by source of first breakoff

	Among interviews with at least 1 breakoff
2011 Production characteristics	A Respondent	B Interviewer	C Telephone	Statistically different pairs
Average interview length (minutes)	102.3	103.9	101.6
Average number of calls	21.0	19.1	18.2	a,c*
% with refusal on any call	6.8	3.8	4.2	a,b*
% requiring tracking	23.7	27.7	30.3	a,c*
% using cell phone on any call	78.7	75.6	85.6	a,c**; b,c**
n	1124	397	353

Indicates statistically significant difference at *p<=.05 or **p<=.01 using a t-test.

4.3 Location of first breakoff in the instrument

Table 5 lists the section of the 2011 instrument in which the first breakoff occurred. Column A shows the number of first breakoffs in each section, and Column B shows the percentage of breakoffs within each section as a percentage of the total number of breakoffs. These results show that the instrument sections with the highest overall rates of first breakoff are “Employment & Residence”, accounting for 27.9% of all first breakoffs, followed by “Household Roster”, at 13.1%, followed by “Health”, at 11.9%. The sections on expenditures and income in the past year also have relatively high breakoff rates, each accounting for more than 11% of all first breakoffs. Sections with relatively low rates include “Housing”, “Income two years ago”, “Wealth”, “Pensions”, and “Philanthropy”. Few first breakoffs occur in the sections administered only to families with a newly eligible individual (“Marriages and Births”) or families new to the study (“New Family information”), due at least in part to the few families who receive this section.

Table 5.

Distribution of breakoff by instrument section

	A	B	C	D
Section	Number of breakoffs	Breakoffs by section (%)	Section length of total instrument (%)	Ratio of breakoff rate/ section length rate
Household Roster	248	13.1	4.6	2.85
Housing	110	5.8	7.0	0.83
Employment & Residence calendar	527	27.9	26.0	1.07
Expenditures	216	11.4	10.8	1.06
Income last year	211	11.2	9.4	1.19
Income two years ago	65	3.4	2.1	1.67
Wealth	119	6.3	6.9	0.92
Pensions	98	5.2	4.4	1.17
Health	225	11.9	19.6	0.61
Marriages and Births	25	1.3	1.3	1.00
New Family information	25	1.3	2.7	0.50
Philanthropy	20	1.1	5.3	0.20
Total	1,889	100.0	100.0	1.00

The risk of first breakoff within a section is related to the amount of time each section takes to administer. Column C shows the percentage of average time within each section as a percentage of the total average length of the instrument. A first breakoff rate was then calculated adjusting for the average number of minutes each section takes to administer (Column D). Column D lists the ratio of a section’s absolute first breakoff rate (Column B) to its percentage of average time in the overall instrument (Column C). This ratio facilitates the interpretation of the within-section breakoff rate. A ratio above 1.0 indicates that the number of interviews with first breakoffs is high relative to the average time it takes to administer the section; a ratio below 1.0 indicates that the number of interviews with first breakoffs is low relative to the length of the section.

Examination of Column D shows that while the sheer number of interviews with first breakoffs is most prevalent in sections of “Employment & Residence”, “Household Roster” and “Health”, their comparative rankings are lower when adjusted for their relative lengths in the overall instrument. Adjusted for relative length, the sections with the greatest number of breakoffs are “Household Roster”, “Income two years ago”, “Income last year”, followed by “Pensions”, and “Employment & Residence”.

As shown in Table 6, the distribution of the reason for first breakoff varies by section of the instrument. Respondent-initiated breakoffs are most prevalent in the employment and residential history section, followed by the sections on health, expenditures, and income last year. All of these sections are characterized by questions requiring recall over two years and mental calculations. Nearly 40% of the interviewer-initiated breakoffs occur in the household roster section of the instrument and are also high within the employment and residential history section, which as described above, includes a calendar that differs substantially in design and navigation from the rest of the instrument. Inter- estingly, instead of being evenly distributed throughout the instrument, or increasing over the length of the instrument as might be expected with the depletion of cell phone batteries, breakoffs due to telephone problems cluster in the same sections of the instrument as breakoffs initiated by respondents.

Table 6.

Distribution of breakoff (%) by instrument section by source (n=1,889)

	Reason
Section	Respondent	Interviewer	Telephone problem
Household Roster	5.3	39.6	7.9
Housing	4.2	8.4	8.2
Employment & Residence calendar	30.5	21.5	26.9
Expenditures	12.5	8.2	11.6
Income last year	12.3	7.2	12.2
Income two years ago	3.8	1.5	4.5
Wealth	7.7	3.7	4.8
Pensions	6.7	2.0	4.0
Health	13.5	6.2	13.3
Marriages and Births	1.6	0.7	1.1
New Family information	1.2	0.7	2.3
Philanthropy	0.7	0.2	3.1
n	1,132	404	353

4.4 The role of “introductory” questions

Consistent with prior research finding evidence of the heaping of breakoffs on questions that introduce a new section and signal a transition to a different content area (Groves and Kahn 1979; Peytchev 2009), breakoffs in the current study are also high on introductory questions. Across the complete set of 1,381 questions in the 2011 interview, 2.6% of the questions (n=36) were identified as being ‘introductory’. Of the 1,889 interviews with at least one breakoff, approximately 8.2% occurred on an introductory question. This may be a substantial underestimate of the true number of cases that broke off on introductory questions, as limitations of the paradata did not permit inclusion of breakoffs that may have occurred on the introduction to the event history calendar. The fact that at least 8.2% of the breakoffs occur on a type of question that represents only 2.6% of the total number underscores the potential of these questions to give respondents an opportunity to deny an implicit request to continue the interview. These introductory questions can be divided into two types: a set of 9 questions that are the very first question in a new section and introduce a completely new topic, and the remaining set of 27 questions which have introductory statements embedded within sections, and thus introduce new but related questions (e.g., introducing the topic of health insurance after asking about health conditions in the “Health” section). Both types of introductory questions had breakoff rates higher than observed on other questions. The questions that introduced completely new sections had a breakoff rate of 3.4%; the set of embedded introductory questions had a slightly higher breakoff rate of 4.8%.

4.5 Predictors of breaking off during the interview – Multivariate models

Multivariate regression models were estimated to examine the effects of characteristics of sample members and the production effort undertaken during 2009 and 2011 on the odds of a breakoff during 2011. Results from the logistic regression model predicting at least one breakoff are shown in Column A of Table 7. Column B presents results of a multinomial logistic regression model examining the differential impact of these characteristics on the source of breakoff. In this model, respondent-based breakoffs are fixed as the group to which the other two breakoff types are compared.

Table 7.

Odd ratios of three multivariate models of sample and production characteristics

	A. Any breakoff vs. none	B1. Respondent- vs. Interviewer-breakoff	B2. Respondent- vs. Telephone-breakoff
Sample member characteristics as of 2009
Change in family composition
No (reference)	1.0	1.0	1.0
Yes	1.3**	1.4**	1.1
Number of individuals in family unit
Less than 3 (reference)	1.0	1.0	1.0
3–12	1.2**	0.9	0.8
Number of jobs held over past two years
1–2 (reference)	1.0	1.0	1.0
3 or more	1.2	0.9	0.4**
Total family income
Bottom three quartiles (reference)	1.0	1.0	1.0
Top quartile	1.0	0.5**	0.6**
Education of head of family
High school or less (reference)	1.0	1.0	1.0
Greater than high school	1.1	0.8*	0.7*
Age of head of family
46 or older (reference)	1.0	1.0	1.0
Less than 46	0.9	0.7*	1.3
Sample types
SRC (reference)	1.0	1.0	1.0
SEO	1.4**	0.9	0.9
Original PSID sample (reference)	1.0	1.0	1.0
Immigrant sample	1.6**	0.7	1.0
Prior wave (2009) production characteristics
At least one breakoff
No (reference)	1.0	1.0	1.0
Yes	2.3**	0.8	0.9
Whether completed interview
Yes (reference)	1.0	1.0	1.0
No	1.6*	0.5	1.2
Number of calls made
Less than 6 (reference)	1.0	1.0	1.0
6 or more	1.3**	0.7*	0.7*
Whether case required tracking
No (reference)	1.0	1.0	1.0
Yes	1.1	1.1	1.1
Whether case ever gave interim refusal
No (reference)	1.0	1.0	1.0
Yes	1.5**	0.8	0.9
Cell phone used on any call
No (reference)	1.0	1.0	1.0
Yes	1.5**	1.0	1.6**
Interview length (percentile)
Below 50th (reference)	1.0	1.0	1.0
50th–75th	1.5**	0.9	1.2
75th–90th	2.2**	1.1	0.9
Above 90th	4.1**	1.1	1.1
New Interviewer
No (reference)	1.0	1.0	1.0
Yes	1.0	1.0	0.8
n	8,311	n=1,889

^**p<=0.01

^*p<=0.05.

Predictors of at least one breakoff

The results of logistic regression models predicting having at least one breakoff compared to having no breakoffs (Column A) show that several characteristics of sample members increase the odds of breakoff, including: being in the low income oversample or immigrant sample, experiencing a change in family composition, and having more than two individuals living in the family unit. Number of jobs, high family income, age and education of household head did not affect the odds of a breakoff.

Several features of prior wave field effort were strongly predictive of current wave breakoff, including prior wave nonresponse, having an interim refusal, and a high number of calls to finalize the interview, effects that demonstrate strong relationships between these types of survey response behaviors. Prior wave breakoff was also a strong predictor of current wave breakoff. Closer examination of this relationship showed considerable inconsistency in the behavior between waves, with 59% of breakoff cases in 2009 not having a breakoff in 2011, and 75% of all breakoff cases in 2011 not having a breakoff in 2009.

As expected, the odds of breakoff were significantly increased by the use of a cell phone, with cell phone users about 1.5 times more likely to have a breakoff. Interview length above the median increased the odds of breakoff in a linear fashion. Compared to interviews that were finalized with lengths at the median or lower, those between the median and the 75^th percentile were 1.5 times more likely to breakoff. Odds of breaking off were more than double compared to those at the median or lower for interview lengths between the 75^th and 90^th percentile, and were four times more likely for those above the 90^th percentile.

Predictors of source of breakoff

Columns B1 and B2 in Table 7 presents results of the multinomial logistic regression model of the source of breakoff. This model examines how sample characteristics and production effort in the prior and current wave predict the differential odds of a breakoff initiated by an interviewer, or due to a telephone problem, compared to being initiated by the respondent. Since respondent-based breakoffs are fixed as the reference category, predictors with ORs less than 1.0 may be interpreted as those that increase the odds of respondent-based breakoffs compared to those due to an interviewer or telephone problem. The results in Column B1 show that several characteristics of sample members differentially increased the odds of a respondent-based breakoff compared to an interviewer-initiated breakoff including: high total family income, high education, and young age of the household head. Interestingly, having a change in family composition, which requires modification by the interviewer of the preloaded household roster, significantly increased the odds of an interviewer-initiated breakoff relative to a respondent breakoff. A high number of calls to finalize the case in the prior wave also predicted greater odds of a respondent-based breakoff compared to an interviewer-initiated breakoff. The use of a cell phone, high interview length, and interviewer experience did not differentially predict respondent breakoffs from those due initiated by interviewers.

As shown in Column B2, several characteristics differentially predict respondent-based breakoffs from those due to telephone problems. As was found for the comparison between respondent breakoffs and interviewer breakoffs (B1), high family income, high education, and a high number of prior wave calls increased the odds of a breakoff being initiated by a respondent compared to a telephone problem. In addition, compared to breakoffs due to telephone problems, having three or more jobs significantly increased the odds of a respondent-based breakoff. Finally, as we would expect, the use of a cell phone increased the odds of a telephone-based breakoff compared to a respondent-based breakoff.

5 Discussion

The goal of this study was to expand the existing literature on survey breakoffs by providing information about the phenomenon in a computer-assisted telephone interview setting. The challenges posed by temporary breakoffs in telephone surveys have been largely unrecognized in the literature. This study shows that in an ongoing U.S. telephone panel survey, interviews with breakoffs are quite prevalent, have correlates that may be traced to respondent, interviewer, and telephone problems as well as features of the instrument, and are associated with substantial field effort. Nearly one quarter of all the interviews in the 2011 wave of the PSID had at least one breakoff, an increase of 65% over the rate in the prior wave. It is not yet known whether this increase between 2009 and 2011 is a unique occurrence associated with particular characteristics of the 2011 PSID field period, or represents a trend that will continue in future waves. A study examining the reasons for the increase in breakoff rate between waves is underway, with initial findings consistent with the results of this study, pointing to an unexpected increase in interview length in combination with increases in cell phone use as key determinants of this change.

The current study documents that interviews with breakoffs are widely distributed across the field interviewer staff. Regardless of experience, virtually all interviewers had at least one, and most had at least one of all types – initiated by themselves, the respondent, or due to telephone problems. The finding that breakoffs were more often initiated by interviewers themselves than due to telephone problems such as cell phone battery depletion was unexpected, and the observation that the vast majority of them clustered in particular sections of the questionnaire highlights the potential for using these data to make modifications to the instrument and/or address training needs.

Field effort for interviews with breakoffs was much higher than for those completed in one session, requiring nearly double the number of calls and greater refusal aversion and tracking effort. These interviews were also longer and more likely to involve the use of cell phones. Most of the field effort as measured by number of calls and timing of refusal aversion and tracking occurred prior to the breakoff. Yet it is noteworthy that the field effort required after the breakoff was also quite substantial, especially for those cases that required tracking or refusal conversion. Together these results paint a picture of respondents who are especially challenging to contact, and once contacted, to complete the interview. Results from multivariate models lend support to this in finding that prior wave breakoff, nonresponse, a high number of calls, and interim refusal were all characteristics of respondents that increased the odds of breakoff in the current wave. As Peytchev (2011) notes, most prior work on the persistence of survey response behavior has focused on describing characteristics of hard-core non-respondents. Understanding the factors related to consistency between waves in other survey behaviors such as break off among respondents who ultimately complete the interview, but at a high use of field resources, is an important topic for future research.

As expected, breakoffs initiated by respondents tended to occur on questions high in cognitive burden that required lengthy recall periods for information and events that may not be especially salient (i.e., detailed income components over the past two years; providing specific dates for work transitions over the past two years), and mental calculations (i.e., summing components of expenses to a total). Interviewers most often broke off in the sections of the instrument that were different from the standard style of questions asked in the other sections that used grids and calendars which may be complex for interviewers to navigate. Together these results indicate that cognitive burden is an underlying source of breakoffs for both respondents and interviewers. As expected, cell phone use was a strong predictor of breakoffs due to telephone problems compared to other types. However, contrary to expectations, telephone breakoffs did not increase over the course of the instrument, as would be the case if battery depletion is a main cause. In fact, the distribution of breakoffs by questionnaire section for telephone problems was very similar to those initiated by respondents, suggesting that at least some breakoffs recorded as telephone problems may in fact be deliberate disconnections by respondents.

Consistent with other research (Groves and Kahn 1979; Peytchev 2009; Tourangeau et al. 2002), high rates of breakoffs occurred on questions that introduced new sections of the instrument. While such questions accounted for less than 3% of all questions in the instrument, they were those on which at least 8% of all breakoffs occurred. Questions that included introductory phrases and described a new but related set of upcoming topics had a higher rate of breakoffs than questions that introduced completely new sections. While these phrases are usually embedded within an interview to facilitate smooth transitions and conversational flow, they may provide a convenient opportunity for respondents to discontinue the interview. In a lengthy interview, such phrases may also foreshadow additional content yet to come – whether correctly or not – leading respondents to reevaluate their decision to continue cooperating with the survey request. This possibility is consistent with results from a web survey experiment finding that respondents who were led to believe that the survey is longer by presenting a progress indicator that changed very little had more than double the risk of breakoff compared to those with no progress indicator (Peytchev 2009). In light of these findings, survey practitioners may want to use such phrases conservatively, especially in a lengthy instrument that has already introduced the content of a new section, and avoid using them near the end of an interview, where their placement may cause respondents to overestimate the remaining interview length.

A number of respondent characteristics in this study – including high total family income, relatively high levels of education, and requiring a high number of calls to finalize the prior wave interview – were significantly differentially predictive of breakoffs initiated by respondents compared to interviewers or due to telephone problems. Drawing on the leverage-saliency theory of survey participation proposed by Groves et al. (2000), it is possible that such characteristics are associated with the erosion of initial positive reactions to the survey request as the interview draws out as additional factors take on greater salience, such as competing time commitments. The leverage of incentive payments may also be lower for individuals at high income levels. Hazard models that predict risk of breakoff from detailed question types and respondent characteristics would help us to better understand how attributes of respondents, such as having many income components or jobs, in combination with exposure to various question types, affect breakoff risk.

As expected, field effort in the prior wave and interview length in the current wave were strong predictors of current wave breakoff, regardless of type. Interview lengths above the median were strong predictors of breaking off net of many socio-demographic characteristics of respondents, indicating that completing a long interview on the telephone in one session may simply be too burdensome for many individuals. Providing respondents with options for completing a lengthy telephone interview, such as deliberately setting specific appointments for multiple sessions – or offering web modules to complete some or all of the interview may alleviate burden, increase perceptions of control (e.g., see Couper 2008), and help manage and conserve field resources.

Consistent with the finding that interviewer-initiated breakoffs occurred most frequently in the household roster section, multivariate analyses showed that these breakoff types were strongly predicted by interviews that included a change in family composition. This finding likely reflects the difficulty interviewers experience in modifying information in the preloaded household roster. Given the high rate of family composition changes each wave – with more than one-third of all respondents reporting at least one – it is clear that additional training resources should be devoted to this section of the instrument, with training examples focusing on how to enter data and navigate this section when a compositional change is reported.

In general, a practical use of the breakoff data is to identify instrument sections that need greater emphasis during training, as well as target specific interviewers who could benefit from additional training. Focusing debriefing calls with interviewers on sections and questions that have high breakoff rates may also provide an efficient way to obtain information about difficulties interviewers face. Moreover, the data can inform instrument development, with scrutiny given to high breakoff questions and sections. Ways to reduce cognitive burden on questions for which breakoff rates are high, such as those involving complex mental calculations or lengthy recall periods, should be explored. For example, simply breaking apart single questions that ask respondents to recall information over two calendar years into two separate questions may reduce burden and facilitate recall.

Future research is needed to specify the characteristics of interview questions that may lead respondents or interviewers to discontinue the interview. The ideal study would systematically code each question in the instrument on various characteristics such as lengthy recall period, mental calculation, sensitivity of the topic, lengthy question, introductory phrase, etc. to quantify the association between question type and breakoff using models that predict the probability of breakoff for each successive item, based on the coded characteristics of the question. Examining the interplay of question types with respondent and interviewer characteristics such as education, socio-economic status, and other sociodemographic attributes would additionally provide useful information that could lead to instrument improvements and the reduction of breakoffs.

An important agenda item for future work is the placement of the breakoff phenomenon within a conceptual framework for survey response behavior. Existing theories of survey nonresponse describe refusal and non-contact as the main components (e.g., Groves and Couper 1998) and abundant research describes various demographic characteristics of refusers and non-contacts (e.g., Lepkowski and Couper 2002; Lynn and Clarke 2002). Yet there is virtually no work that incorporates breakoffs into models of survey participation. Are breakoffs part of a continuum of survey response behavior along with response, reluctance, and refusal, or are they better understood as a subcomponent of nonresponse, or are they a unique response behavior altogether? The current study shows that nonresponse in the prior wave is a strong predictor of current wave breakoff, indicating that the two have shared causes. Interestingly, while prior wave breakoff is also strongly related to current wave breakoff, most of the current wave breakoff cases are new in 2011 and most of the prior wave breakoff cases did not have a breakoff in the current wave. This pattern of results complements the findings by Peytchev (2011) who also finds that nonresponse and breaking off are related to one another, with nonrespondents more likely than respondents to breakoff in subsequent surveys, and that there is greater consistency in nonresponse than in breaking off. These results and those of the current study underscore the need for the continued study of the relationship between breakoff behavior and unit nonresponse.

Finally, it is worth noting that this study was possible because of the collection of paradata on interview breakoffs. As others have noted, paradata have an enormous and largely untapped potential for providing a clearer understanding of many aspects of survey design and management (Couper 1998; 2009; Nicolaas 2011; 2012). These data have been largely underutilized because of their complexity, in part the result of their incidental collection by computerized systems designed with the collection of the survey interview data as the main goal. Systematic and planful collection of paradata would help advance various survey methodological investigations. In particular, deliberate attention to the best ways to collect paradata on interview breakoffs would provide a clearer understanding of their causes and consequences, expanding theoretical frameworks of survey participation.