A 2-1-1 Research Collaboration

Abstract

Background

2-1-1 serves as a lifeline in times of crises. These crises often cause a spike in call volume that can challenge 2-1-1’s ability to meet their service quality standards. For researchers gathering data through 2-1-1s, a sudden increase in call volume might reduce accrual as 2-1-1 has less time to administer study protocols. Research activities imbedded in 2-1-1 systems may directly affect 2-1-1 service quality indicators.

Purpose

Using data from a 2-1-1 research collaboration, this paper examines the impact of crises on call volume to 2-1-1, how call volume affects research participant accrual through 2-1-1, and how research recruitment efforts affect 2-1-1 service quality indicators.

Methods

t-tests were used to examine the effect of call volume on research participant accrual. Linear and logistic regressions were used to examine the effect of research participant accrual on 2-1-1 service quality indicators. Data were collected June 2010–December 2011; data were analyzed in 2012.

Results

Findings from this collaboration suggest that crises causing spikes in call volume adversely affect 2-1-1 service quality indicators as well as accrual of research participants. Administering a brief (2–3 minute) health risk assessment did not negatively affect service quality, but administering a longer (15–18 minute) survey had a modest adverse effect on these indicators.

Conclusions

In 2-1-1 research collaborations, both partners need to understand the dynamic relationship between call volume, research accrual, and service quality, and adjust expectations accordingly. If research goals include administering a longer survey, increased staffing of 2-1-1 call centers may be needed to avoid compromising service quality.

Introduction

Throughout this supplement to the American Journal of Preventive Medicine (AJPM), studies have documented the many ways that 2-1-1 serves as a lifeline in times of crises such as hurricanes, blackouts, disease outbreaks and foreclosure.^1–3 Although the exact timing and nature of such events is unpredictable, their occurrence in some form is largely expected by 2-1-1s. Their effect is generally a dramatic, if temporary, increase in call volume.

For 2-1-1s, any sharp rise in call volume might make it harder to meet their service quality standards, namely a short waiting time for each call to be answered (i.e., “time in queue”) and a low rate of callers hanging up before their call is answered (i.e., “abandon rate”). For researchers gathering data through 2-1-1s, a sudden increase in call volume might reduce accrual as information specialists have less time to administer study protocols. Even less is known about the extent to which research activities imbedded in 2-1-1 systems might directly affect 2-1-1 service quality indicators. The purpose of this paper is to evaluate, for 2-1-1s, the impact of crises on call volume, how call volume affects service quality indicators and research participant accrual by information specialists, and how research participant accrual efforts affect service quality indicators.

Methods

Study Setting and Description

United Way 2-1-1 Missouri (hereafter, 2-1-1 Missouri) serves 99 of 114 counties in the state, excluding 15 counties in the greater Kansas City area that are served by another 2-1-1 system. 2-1-1 Missouri received nearly 166,000 calls in 2011. Data included in this analysis were collected during an 18-month period (June 10, 2010 to December 23rd, 2011) during which recruitment was taking place for a 2-1-1 research study. Recruitment into the study occurred on weekdays only, when 94% of calls are received. United Way 2-1-1 Missouri callers are primarily low-income, disproportionately women and minorities, and generally are seeking assistance with basic human needs.^4,5

Research participant accrual data in this study come from an ongoing trial conducted in collaboration with 2-1-1 Missouri. In the project, 2-1-1 information and referral specialists (specialists) administer a brief (2–3 minute) risk assessment of six cancer prevention and screening behaviors (colonoscopy, mammography, Pap, HPV vaccination, smoking cessation, and smokefree rules for the home) to callers after they have received standard service. Where indicated by the results of the risk assessment, referrals are offered to free or low-cost programs that provide a needed service. Callers that have at least one cancer control need are invited to participate in an intervention trial, and those that accept are administered a longer (15–18 minute) baseline survey while still on the phone with the 2-1-1 specialist.

Two specialists from 2-1-1 Missouri are responsible for recruiting callers into the study and administering the risk assessment and baseline survey. They account for approximately 12% of specialist staffing on a given shift. Calls coming into the 2-1-1 exchange enter a central phone system and are directed in the order received to the specialist who has been idle longest (i.e., off the phone and available) or the first to become available if all are engaged at the time of the call. This system ensures that the assignment of any call to any specialist, including the study-dedicated staff, is random.

After providing standard 2-1-1 service, specialists offer eligible callers the opportunity to complete the brief risk assessment. Participants recruited on 4 randomly selected days per week were assigned to one of three intervention conditions and participants recruited on 1 day per week were assigned to a control group. Because the control group assessment and protocol took less time to administer, study condition (intervention vs control) is accounted for in the analyses described below. For more detail on the study, including intervention conditions, see Kreuter et al.⁵ in this AJPM supplement.

Measures

In this paper, an examination is made of the association between call volume, research participant accrual, and service quality indicators. The current study reports on data from June 15, 2010 through December 30, 2011. Data were analyzed in 2012 using SPSS 19.0.

Call volume

Call volume includes all calls coming in to 2-1-1 per day through the Cisco phone system used in the call center. More than 700 calls a day is considered “very high” call volume at 2-1-1 Missouri, causing service quality indicators to be affected. For regression analyses, an additional variable was created to categorize call volume into gradations of 100 calls, for more meaningful interpretation of results.

Research participant accrual

A study database records the number of risk assessments and baseline surveys completed each day. Risk assessments include 6–20 questions (based on age and gender) and take 2.5–4 minutes to complete. Baseline surveys include 39 questions and take 14–18 minutes to complete. All participants who complete a baseline survey have also completed a risk assessment. Those who completed a risk assessment only did not complete a baseline survey.

Service quality indicators

Service quality indicator data in clude average time in queue, abandon rate, and service level. Each is calculated daily. Average time in queue is calculated for all calls received in a day and represents the mean period (in seconds) a call spends in queue (i.e., waiting) before it is handled by a specialist. The goal is to keep this number as low as possible; for this analysis a goal of ≤60 seconds was used. Abandon rate is the percentage of total calls abandoned within 60 seconds of entering the queue. The goal for the abandon rate is <10%.

Service level is the percentage of incoming calls that are answered by a specialist within 60 seconds of entering the call queue. The service level goal for 2-1-1 Missouri is 80% of calls answered within 60 seconds of entering queue. Each quality indicator variable is calculated by the Cisco phone system, and 2-1-1 Missouri staff create daily data reports to share with the call center staff and the study team. For all three service quality indicators, data from Saturdays and Sundays are excluded because research participant accrual occurred on weekdays only. Of 399 weekdays, data on call volume and service quality indicators were unavailable for 8 days (2%). Service quality indicators, goals, and sources of data for all variables used in this analysis are represented in Table 1.

Table 1.

Variables used in call volume, recruitment and quality indicator analysis

Variable	Description	Goal for study	Source
Call volume	Number of calls that enter the Cisco phone system, including abandoned calls	n/a; >700 per day considered “very high”	Cisco, 2-1-1
Average time in queue	Average time that a call sits in queue before being answered	≤60 seconds	Cisco, 2-1-1
Abandon rate %	Percentage of total calls that are abandoned in less than 60 seconds	<10%	Cisco, 2-1-1
Service Level	Percentage of calls answered within 60 seconds	80%/60 seconds	Cisco, 2-1-1
Risk assessments completed	Number of risk assessments completed per day by study- dedicated information specialists		Study database
Baseline surveys completed	Number of baseline surveys completed per day by study- dedicated information specialists		Study database

Statistical Analyses

The primary analytic goals of the study were to determine: (1) the effect of 2-1-1 call volume on research participant accrual, and (2) the effect of call volume and research participant accrual on 2-1-1 service quality indicators. An additional goal was to explore visually the relationship between specific crisis events that occurred during the observation period and call volume to 2-1-1. To examine the effect of call volume on research participant accrual, a t-test was performed to determine the difference in number of risk assessments and baselines completed between standard call volume days and very high call volume days.

To examine the effect of research participant accrual on 2-1-1 service quality indicators, two sets of analyses were conducted. First, linear regression was used to model each quality indicator (service level, abandon rate, average time in queue). Second, binomial logistic regression was used to model not meeting quality indicator goals. All regression models included the same three independent variables: call volume (in 100-call segments); number of risk assessments only completed; and number of baseline surveys completed (which include a risk assessment).

Because callers recruited into the control group had shorter call times than those recruited into an intervention group, all regression models controlled for type of recruitment day (intervention/control). Control recruitment days served as the referent in all models. Parameter estimates and 95% CIs for full models are reported.

Results

Descriptives

The number of completed risk assessments per day ranged from 0 to 32, with a median of 7 (M= 7.5, SD=6.3); completed baseline surveys per day ranged from 0 to 12, with a median of 3 (M=3.2, SD=2.9); daily call volume ranged from 26 to 2390, with a median of 554 (M=601, SD=279). For service quality indicators, the lowest service level observed in a day (% of total calls answered within 60 seconds) was 5%, and the highest was 100%; the median was 56% (M=54%, SD=25). Abandon rate per day ranged from 0% to 73%, with a median of 9% (M=14%, SD=13). Finally, average time in queue per day (in seconds) ranged from 2 to 650, with a median of 79 (M=115 seconds, SD=108).

Call Volume and Recruitment

Figure 1 and Table 2 show call volume, risk assessments and baseline surveys completed for the period of June 15, 2010 to December 23, 2011. Each set of bars represents 2 weeks of data, collected on every weekday for that time period. As call volume increases, recruitment into the study, including both risk assessments and baseline surveys completed, decreases. Spikes in call volume co-occur with crises such as heat waves and natural disasters. During these periods, completion of risk assessments and baseline surveys drops to zero or near-zero levels.

Figure 1.

Biweekly risk assessment and baseline data collection by call volume, for 2-1-1 Missouri, 6/15/10 to 12/23/11

Table 2.

Biweekly risk assessments, baseline surveys, and call volume, 6/15/10 to 12/30/11

Time period	Biweekly Risk Assessments	Biweekly Baselines	Biweekly Call Volume
6/15/10 - 6/25/10	154	69	4698
6/28/10 - 7/9/10	140	62	4332
7/12/10 - 7/23/10	107	52	5655
7/26/10 - 8/6/10	35	23	6683
8/9/10 - 8/20/10	27	10	9519
8/23/10 - 9/3/10	36	15	9902
9/6/10 - 9/17/10	50	17	9618
9/20/10 - 10/1/10	92	36	8301
10/4/10 - 10/15/10	97	33	6733
10/18/10 - 10/29/10	90	38	5298
11/1/10 - 11/12/10	107	43	5290
11/15/10 - 11/26/10	64	36	4015
11/29/10 - 12/10/10	110	49	4533
12/13/10 - 12/24/10	43	16	3898
12/27/10 - 1/7/11	49	24	3330
1/10/11 - 1/21/11	52	24	3609
1/24/11 - 2/4/11	33	8	4380
2/7/11 - 2/18/11	51	18	3496
2/21/11 - 3/4/11	69	33	3169
3/7/11 - 3/18/11	85	33	3290
3/21/11 - 4/1/11	108	44	3642
4/4/11 - 4/15/11	124	54	4462
4/18/11 - 4/29/11	44	16	6108
5/2/11 - 5/13/11	1	1	6924
5/16/11 - 5/27/11	20	9	10533
5/30/11 - 6/10/11	0	0	7546
6/13/11 - 6/24/11	99	45	5496
6/27/11 - 7/8/11	98	50	5243
7/11/11 - 7/22/11	98	43	8481
7/25/11 - 8/5/11	41	23	7280
8/8/11 - 8/19/11	141	56	6226
8/22/11 - 9/2/11	64	36	7266
9/5/11 - 9/16/11	66	32	7908
9/19/11 - 9/30/11	46	21	7715
10/3/11 - 10/14/11	23	6	8138
10/17/11 - 10/28/11	21	9	6164
10/31/11 - 11/11/11	43	16	5392
11/14/11 - 11/25/11	92	49	4201
11/28/11 - 12/9/11	207	73	5495
12/12/11 - 12/23/11	159	60	5053
12/26/11 - 12/30/11	26	10	1622
Total	3012	1292	240644

Similarly, when call volume is dichotomized into lower (<700 calls a day) and higher (≥700 calls) categories, its impact on research recruitment is evident. The mean number of risk assessments completed during lower–call volume days is 8.4 (6.0) compared to 5.1 (6.2) on higher–call volume days (p<0.001). Likewise, the number of baseline surveys completed drops from 3.6 (2.9) on days with lower call volume to 2.2 (2.9) on higher–call volume days (p<0.001). During the study period, 27% of days had very high call volume.

Call Volume, Recruitment, and Service Quality Indicators

Linear regression found call volume and number of completed baseline surveys to be associated with each of the service quality indicators (Table 3). A 100-call increase in daily call volume decreased service level by 6.8%, increased abandon rate by 3.3%, and increased average time in queue by 22.7 seconds. Similar trends were found for each completed baseline survey in a day, which decreased that day’s service level by 1.07%, increased abandon rate by 0.48%, and increased the average time in queue by 5 seconds. There was no association between the number of risk assessments only completed and any of the three service quality indicators.

Table 3.

Results of linear regression of call volume and recruitment on service quality indicators

	β	95% CI	p-value
Predicting service level
Call volume (per 100 calls)	−6.79	−7.39, −6.19	<0.001
Risk assessments only	0.39	−0.16, 0.93	0.166
Baseline surveys	−1.07	−1.81, −0.33	0.005
Intervention recruitment day	−4.85	−9.39, −0.32	0.036

Predicting abandon rate
Call volume (per 100 calls)	3.26	2.91, 3.60	<0.001
Risk assessments only	−0.21	−0.52, 0.10	0.184
Baseline surveys	0.48	0.05, 0.90	0.027
Intervention recruitment day	2.95	0.36, 5.55	0.026

Predicting average time in queue
Call volume (per 100 calls)	22.73	19.61, 25.86	<0.001
Risk assessments only	−2.74	−5.59, 0.11	0.059
Baseline surveys	4.99	1.19, 8.87	0.012
Intervention recruitment day	27.65	3.99, 51.31	0.022

In logistic regression models (Table 4), only call volume was associated with not meeting service quality indicator goals. Each 100-call increase in call volume increased the odds of not meeting the service level goal by 3.0%, not meeting the abandon rate standard by 2.5%, and not meeting the average time in queue goal by 2.2%. Neither the number of risk assessments only completed nor number of baseline surveys completed was associated with not meeting service quality indicator goals.

Table 4.

Results of logistic regression of call volume and recruitment on meeting dichotomous quality indicator goals

Outcome = Did not meet service-level standard (≥80% answered in 60 seconds)	AOR	95% CI	p-value
Call volume (per 100 calls)	2.95	2.23, 3.89	<0.001
Risk assessments only	1.04	0.94, 1.16	0.431
Baseline surveys	1.09	0.94, 1.27	0.269
Intervention recruitment day	1.12	0.45, 2.79	0.807

Outcome = Did not meet abandon rate standard (<10%)
Call volume (per 100 calls)	2.53	2.07, 3.09	<0.001
Risk assessments only	0.98	0.90, 1.07	0.695
Baseline surveys	1.01	0.90, 1.13	0.890
Intervention recruitment day	1.38	0.68, 2.80	0.378

Outcome = Did not meet average time in queue goal (≤60 seconds)
Call volume (per 100 calls)	2.17	1.82, 2.58	<0.001
Risk assessments only	0.98	0.90, 1.06	0.548
Baseline surveys	1.07	0.96, 1.20	0.207
Intervention recruitment day	1.79	0.91, 3.52	0.093

Discussion

United Way 2-1-1 Missouri strives to provide high-quality service to its callers. Very high call volume to 2-1-1 Missouri has the greatest negative impact on providing the best service to 2-1-1 callers. In the current analyses, call volume was the only significant predictor of not meeting quality indicator goals, although both call volume and the number of baseline surveys completed had a negative impact on service quality indicators. Although standard call volume (<700 calls a day) allowed for reasonable recruitment of 2-1-1 callers into a research study, when daily call volume exceeded 700, there were significant drops in recruitment numbers, with some weeks of no recruitment at all.

During the period observed in the current analyses, call volume periodically spiked to over 700 calls per day (or about 7000 per 2-week period; Figure 1) during times of crises for Missourians. The first spike came during a heat wave in the summer of 2010, as callers sought relief from the heat through utility assistance, air conditioning units and fans, and cooling stations. The second large spike began with the “Good Friday Tornado” on April 22, 2011, which caused damage to homes and businesses across the St. Louis area,⁸ and was followed shortly thereafter by extensive flooding in Missouri through May. Call volume peaked the day of the Joplin MO tornado on May 22, 2011,⁹ which killed 162 people and devastated the Joplin area. During these crises, 2-1-1 Missouri acted as a primary source of information for victims, loved ones, and concerned citizens, and served as an intake center for enrolling Joplin tornado victims into Federal Emergency Management Agency assistance.

The study team shared 2-1-1’s commitment to improving the health and lives of the underserved and understood that research participant accrual could not take priority over providing high-quality service and meeting basic human needs. Research teams interested in partnering with 2-1-1 systems must be cognizant of 2-1-1’s mission, recognize that it is a service—not a research—organization, and plan in advance for how recruitment might be handled in times of crises. Likewise, 2-1-1 systems should make their quality assurance goals clear to potential research partners, and indicate at what point those goals would take priority over research participant accrual.

Although the current study conducts both a brief cancer risk assessment and a longer baseline survey, the latter is for research purposes only. The dissemination goal for integrating proactive health screening and referrals into all 2-1-1 systems would require only administering the brief risk assessment. The current analyses found that administering the risk assessment alone had no adverse effect on service quality indicators.

For researchers administering longer surveys through 2-1-1, it may be necessary to adjust staffing of 2-1-1 call centers to avoid negatively affecting service quality indicators. For instance, using the data from the current sample, the impact of administering just ten 20-minute surveys in a day during which an event occurs that has caused call volume to spike by 300 calls could potentially decrease that day’s service level by 28%, increased abandon rate by 13%, and increase the average time in queue by nearly 2 minutes. Even if that day started with perfect service quality indicator scores—100% service level, 0% abandon rate, and 0 seconds in queue—the impact of both call volume and recruitment can drop these service quality indicators below acceptable levels for this 2-1-1 system.

A key limitation of this study is that the accrual of research participants involved only a fraction of 2-1-1 Missouri’s information and referral specialists. Had they all been administering risk assessments and baseline surveys, it seems likely that the impact on service quality indicators would have been greater. Likewise, a larger detrimental effect of increased call volume on research participant accrual would also be expected under such conditions.

Because different 2-1-1 systems have different goals and standards for quality service, it is possible that tolerance for the demands of research participation might also vary by 2-1-1s. The three indicators included in this study were those identified as most important by 2-1-1 Missouri. Every 2-1-1 has some version of these, as does the Alliance of Information and Referral Systems (AIRS), a North American professional membership organization for information and referral delivery systems.^6,7 But individual 2-1-1 systems should view these findings in light of their own standards.

In 2-1-1 collaborations with research studies, both partners need to understand the dynamic relationship among call volume, research accrual, and service quality, and then adjust expectations accordingly. Research teams may take this into consideration when designing recruitment tools and interventions partnering with 2-1-1 systems. In addition, 2-1-1 systems interested in establishing a research partnership can see that reasonable recruitment or intervention strategies should have minimal impact on quality indicators for service to its callers. Although the current collaborative team of researchers and 2-1-1 staff learned these lessons through experience and as the project progressed, it is hoped that new 2-1-1 and research partners will benefit by applying this knowledge in the planning stages of new projects.