An Evaluation of the Impact of Using an Alternate Caller ID Display in the National Immunization Survey

Abstract

The National Immunization Surveys (NIS) include dual frame random-digit-dial telephone surveys used to monitor vaccination coverage in the United States among children age 19–35 months (NIS-Child) and adolescents age 13–17 years (NIS-Teen), and to assess influenza vaccination for children age 6 months-17 years (NIS-Flu). The surveys collect household-reported demographic and access-to-care data during telephone interviews with the survey-eligible child’s parent or guardian. The parent or guardian is then asked for consent to contact the child’s vaccination provider(s) to obtain a provider-reported immunization history using a mailed questionnaire.

The success of the NIS relies heavily on getting a respondent to answer the telephone, and the caller ID display is the earliest opportunity to convey information to a respondent about the identity of the caller. An evaluation was conducted in Quarter 4 of 2017 to determine the impact on contact rates of using an alternate caller ID display. The caller ID for the NIS surveys was previously set to display “NORC UCHICAGO”, identifying the contractor administering the surveys, with a Chicago-based telephone number. It was hypothesized that having the caller ID display the name of the more recognizable survey sponsor instead of the contractor would increase contact rates. Half of the sample was randomly flagged to display the “NORC UCHICAGO” caller ID text as a control, and the other half was flagged to display “CDC NATL IMMUN” as a treatment. This paper presents the study design, results, conclusions, limitations, and recommendations for future research.

1. Introduction

Success in telephone surveys relies on respondents answering the telephone when called. Caller ID technology is one way to inform the potential respondent about the identity of the caller prior to having answered the telephone, serving as a “business card” and possibly triggering “brand awareness” (Callegaro et al., 2005; Tuckel, 2001).

“Caller ID” typically consists of two parts: the caller ID (CID) which displays the telephone number of the originating call, and the calling party name (calling name) which displays the name associated with the calling party’s telephone number. There are two types of caller ID in the United States: Single Data Message Format, which only displays the CID, and Multiple Data Message Format, displaying both the calling name and CID. Landline telephones typically receive both the CID and calling name only if equipped with caller ID technology, whereas almost all cellular telephones typically receive the CID but not the calling name unless subscribed to a service. In 2006, it was estimated that about 60% of landline telephones had caller ID (AAPOR, 2010); the rise of Voice over Internet Protocol (VoIP) usage for landline telephones has likely only increased the incidence of caller ID on landlines.

The Federal Communications Commission’s “Truth in Caller ID Act” guidelines (as well as industry ethics) impose limits on what telemarketers and researchers can display on the caller ID (FCC, 2018). Calling names must not intend to “defraud, cause harm, or wrongfully obtain anything of value,” and should typically be limited to the use of the name of the call center or the client (Insights Association, 2015). Several studies have examined the impact of using various calling names and CIDs with different area codes. Additionally, some have shown that displaying a calling name vs. no name or “UNKNOWN” can yield higher survey response rates (Matos and Krey, 2005; Trussell and Lavrakas, 2005; Barron and Khare, 2008). However, most of these studies were conducted primarily on landline phones and are now several years old.

NORC at the University of Chicago conducted a study on a large scale, dual-frame random-digit-dial (RDD) telephone survey in 2017 to assess the impact of using an alternate calling name display on response rates and data collection costs for both landline and cell-phone samples.

2. Design of the Experiment

The National Immunization Surveys (NIS), sponsored by the Centers for Disease Control and Prevention (CDC), are RDD telephone surveys that are conducted annually. The surveys provide U.S. national, state, and select local area estimates of vaccination coverage among young children 19–35 months of age (NIS-Child) and adolescents 13–17 years of age (NIS-Teen), and assess influenza vaccination coverage estimates among children 6 months-17 years of age (NIS-Flu). Sampled telephone numbers are dialed and screened to identify households with children in the target age ranges. If the household has an eligible child, information on receipt of influenza vaccination and socio-demographic characteristics are collected; for NIS-Child and NIS-Teen, consent to contact the child’s vaccination provider(s) is also collected. The NIS-Child and NIS-Teen RDD surveys are followed by mail surveys of nominated vaccination providers from the RDD phase to obtain the vaccination histories of the selected children.

From 2011 through 2017, the NIS was a dual-frame RDD survey, sampling from both the landline and cell-phone frames. Prior to the experiment, the caller ID was set to identify the survey contractor, displaying a 312 area code telephone number as the CID and “NORC UCHICAGO” as the calling name for all outbound calls in both the landline and cell-phone samples. In selecting the alternate calling name for this experiment, several elements were considered, including the use of the sponsor name, survey name, and the words “immunization” or “vaccine.” Given the 15 character limitation of the calling name displayed on the caller ID, the calling name “CDC NATL IMMUN” with a different 312 area code number as the CID, was selected to serve as the treatment condition because it referenced both the sponsor name and the study name.

A randomized experiment was conducted on the NIS sample to evaluate the impact of the new calling name. Upon review and approval from the ERB, the experiment began November 12, 2017 and continued through the end of data collection for the fourth quarter of 2017 on January 26, 2018. Half of the sample was set to receive “CDC NATL IMMUN” and the remainder of the sample was set to receive the standard “NORC UCHICAGO” calling name display. Several key outcomes were examined:

• Yield of screener completes among dialed cases

• Yield of interview completes among dialed cases

• Contact rate among dialed cases

• Screener completion rate among contacts

• Survey eligibility rate among screener completes

• Interview completes among identified eligible respondents

• Relative cost per screener complete

• Relative cost per interview complete

Outcomes were assessed separately for landline and cell-phone sample due to the differences in how caller ID is typically displayed on the respective devices.

3. Results

A total of 554,099 landline cases and 2,539,397 cell-phone cases were randomly assigned to either the treatment condition (“CDC NATL IMMUN”) or the control condition (“NORC UCHICAGO”). The yield rates of screener completes for landline and cell-phone sample and their 95% confidence intervals are presented in Figure 1. A significantly higher screener yield rate is observed for the “CDC NATL IMMUN” condition in both sample types (p<0.001), but the effect size is much larger in the landline sample (9.1 vs. 7.9, or a 15% increase) than in the cell-phone sample (5.9 vs. 5.8, or a 2% increase). Although they were not statistically significant findings, the higher screener yield rates for the “CDC NATL IMMUN” condition lead to higher interview yield rates for this condition in both sample types (p=0.032 in the landline sample; p=0.088 in the cell-phone sample), as shown in Figure 2, with the size of the difference again much larger in the landline sample than in the cell-phone sample.

Figure 1:

Yield Rate of Screener Completes Among Dialed Cases

Figure 2:

Yield Rate of Interview Completes Among Dialed Cases

Figures 3 and 4 break the overall yield rate into components (the contact rate among dialed, the screener completion rate among contacts, the eligibility rate among screened, and the interview completion rate among eligibles) for the landline sample and the cell-phone sample, respectively. As seen in Figure 3, while the contact rate in the landline sample was slightly higher for the “CDC NATL IMMUN” condition compared to the control condition (21.6 vs. 21.2, p=0.001), the higher overall yield rate for “CDC NATL IMMUN” condition was driven primarily by a higher screener completion rate among contacts (42.0 vs. 37.3, p<0.001). Statistically significant differences were not observed in the eligibility rate among screener completes or the interview completion rate among eligibles between the conditions in the landline sample.

Figure 3:

Component Rates by Treatment, Landline

Figure 4:

Component Rates by Treatment, Cell-Phone

In the cell-phone sample (Figure 4), while differences were observed between treatment and control for both the contact rate (26.3 vs. 26.0, p<0.001) and the screener rate among contacts (22.4 vs. 22.2, p=0.026), unlike in the landline sample the difference in the screener completion rate among contacts between conditions was small in magnitude. As in the landline sample, statistically significant differences were not observed in the eligibility rate among screener completes or the interview completion rate among eligible.

Finally, relative data collection costs are examined in Figure 5. In the landline sample, the higher yield rate for the “CDC NATL IMMUN” condition resulted in a 14% lower cost per screener complete and a 22% lower cost per interview complete compared to the “NORC UCHICAGO” condition. In the cell-phone sample, because yield rate differences between conditions were small, only modest differences in the cost per screener complete (1%) and the cost per interview complete (4%) were observed between the treatment and control conditions.

Figure 5:

Relative Data Collection Costs by Treatment Condition

4. Discussion

The “CDC NATL IMMUN” calling name condition resulted in a higher yield rate and a lower cost per screener complete when compared to the “NORC UCHICAGO” condition in both sample types. However, the impact of “CDC NATL IMMUN” was greater in the landline sample than the cell-phone sample, with the yield rate of screener completes for the “CDC NATL IMMUN” being 15% higher in the landline sample but only 2% higher in the cell-phone sample; and the cost per screener complete being 14% lower in the landline sample but only 1% lower in the cell-phone sample. This difference in effect by sample type was anticipated because both the calling name and the CID are typically displayed on landline phones, whereas only the CID without the calling name is displayed on cell-phones unless the user is subscribed to a service to display the calling name. While it is unknown what percentage of cell-phone users are subscribed to such a service, the number is likely quite low, as the service is typically provided only for an additional monthly fee.

Prior to conducting the experiment, it was hypothesized that the “CDC NATL IMMUN” calling name would primarily impact the contact rate, i.e., the rate at which respondents answer the phone. While statistically higher contact rates for the “CDC NATL IMMUN” treatment were observed for both sample types, the largest effect was seen not in the contact rate but in the screener completion rate among contacts in the landline sample. Landline respondents that received the “CDC NATL IMMUN” calling name were slightly more likely to answer the phone, and were much more likely to stay on the phone and complete the screener. This is an unexpected result, and it may suggest that the “CDC NATL IMMUN” calling name resulted in a higher yield rate than the “NORC UCHICAGO” calling name largely due to its consistency with the language in the introduction text read to respondents when they answer the phone. The text in the introduction reads:

Landline introduction:

Hello, my name is ___. I’m calling on behalf of the Centers for Disease Control and Prevention. We’re conducting a nationwide immunization survey to find out how many children under 4 years of age are receiving all of the recommended vaccinations for childhood diseases…”

Cell-phone introduction:

“Hello, my name is ___. I’m calling on behalf of the Centers for Disease Control and Prevention. We are conducting a nationwide study with cell phone users regarding childhood immunizations…”

Dillman (2014) emphasizes the importance of consistent communication with survey respondents, and our results may be attributable to that phenomenon. “NORC UCHICAGO” is not referenced in the introduction, but all of the components of the “CDC NATL IMMUN” calling name display appear in the first two sentences of the introduction: “CDC” is consistent with “Centers for Disease Control and Prevention,” “NATL” with “nationwide,” and “IMMUN” with “immunizations.”

5. Limitations

The calling name is not displayed on all devices, and in particular is not displayed on cell-phones that are not subscribed to a caller ID service. While the calling name text to be used can be specified if the calling name is displayed, whether or not it was actually displayed for a particular respondent could not be controlled, and it was not known which respondents had the calling name displayed and which did not. Therefore, estimates of the impact of the “CDC NATL IMMUN” calling name relative to the “NORC UCHICAGO” calling name are estimates of attempting to display these calling names and do not reflect the impact of the calling name, given that it was displayed.

The calling names used in the study, “NORC UCHICAGO” and “CDC NATL IMMUN”, are specific to the NIS, and these conclusions may not be applicable to other calling names or to surveys with a different subject matter, target population, sponsor, or data collection contractor.

6. Future Research

As a result of the study findings, the calling name was set to the “CDC NATL IMMUN” display for all outgoing calls for the NIS beginning in Quarter 1, 2018. For reasons unrelated to the present study, the NIS moved from a dual-frame landline and cell-phone RDD design to a single-frame cell-phone RDD design in Q1 2018.The impact of the “CDC NATL IMMUN” calling name was much smaller in the cell-phone sample than in the landline sample in our experiment. The removal of the landline sample from the NIS limits the expected cost savings to the NIS as a whole resulting from the change to the “CDC NATL IMMUN” display.

As cell-phone applications and built-in device features continue to evolve, it is possible that in the future the calling name will be displayed on a larger proportion of cell phones. Furthermore, with the increasing use of call blocking technologies, the need to concisely and accurately identify oneself as a caller is becoming even more important in survey research (AAPOR 2018). The impact can be two-fold: not only does an accurate calling name mean respondents will better understand who the caller is prior to answering the phone, but it may also allow call blocking services and applications to better distinguish legitimate callers from spammers. If new options become available for the display of the calling name on cell phones, further research will be needed on how to most effectively utilize such options to maximize yield rates for the NIS.