The National Criminal Justice Treatment Practices survey: Multilevel survey methods and procedures

Abstract

The National Criminal Justice Treatment Practices (NCJTP) survey provides a comprehensive inquiry into the nature of programs and services provided to adult and juvenile offenders involved in the justice system in the United States. The multilevel survey design covers topics such as the mission and goals of correctional and treatment programs; organizational climate and culture for providing services; organizational capacity and needs; opinions of administrators and staff regarding rehabilitation, punishment, and services provided to offenders; treatment policies and procedures; and working relationships between correctional and other agencies. The methodology generates national estimates of the availability of programs and services for offenders. This article details the methodology and sampling frame for the NCJTP survey, response rates, and survey procedures. Prevalence estimates of juvenile and adult offenders under correctional control are provided with externally validated comparisons to illustrate the veracity of the methodology. Limitations of the survey methods are also discussed.

1. Introduction

According to the recently released National Survey on Drug Use and Health (2006), the rate of substance abuse or dependence among adult offenders on probation or parole supervision (38.5%) is more than four times that of the general population (9%; Table 7.103B). An analysis of national data on state prison inmates indicates that nearly three fourths are in need of some substance abuse intervention, and 31.5% of male inmates and 52.3% of all female inmates require intensive services such as residential treatment programs (Belenko & Peugh, 2005). Despite these high prevalence rates, relatively little is known about the availability, type, and quality of substance treatment services provided to offenders. There have been no national surveys of treatment across the correctional landscape (prison, jail, probation, and parole), thus leaving a void in our knowledge about current practices and the effectiveness of such practices (Simpson, 2002).

In the last 30 years, three prospective longitudinal surveys—Drug Abuse Reporting Program (DARP) (1969–1973), Treatment Outcome Prospective Study (TOPS) (1979–1981), and Drug Abuse Treatment Outcome Studies (DATOS) (1989–1993)—have substantially enhanced our understanding of the effectiveness of community-based drug treatment services for the general population of substance abusers. In contrast, research on programs and services for offenders is dominated by individual studies that present a collage of findings and only small glimmers of insight into the characteristics of effective programs. Examples include residential drug treatment programs (Hiller, Knight, Broome, & Simpson, 1998; Inciardi, 1999; Knight, Simpson, & Hiller, 1999; Martin, Butzin, Saum, & Inciardi, 1999; Pearson & Lipton, 1999; Simpson, Wexler, & Inciardi, 1999; Wexler & Melnick, 1999), drug courts (Belenko, 2002; Marlowe, DeMatteo, & Festinger, 2003; Marlowe, Festinger, Lee, Dugosh, & Benasutti, 2006; Taxman & Bouffard, 2002, 2004; Wenzel, Longshore, Turner, & Ridgley, 2001), and community supervision (Taxman, 2002; Taxman & Thanner, 2006; Thanner & Taxman, 2003; Sherman et al., 1997). The collage is incomplete, and our knowledge of effective offender treatment, particularly in system-related issues, lags far behind the general treatment field (Farabee, Prendergast, & Anglin, 1998). This is partly due to the relative lack of attention to justice-based treatment studies; it is also attributable to the inherent complexities of the justice system.

As part of the Criminal Justice Drug Abuse Treatment Studies (CJ-DATS) research cooperative, the National Institute on Drug Abuse [NIDA] (1999) sponsored the first comprehensive survey of treatment services within correctional agencies. The National Criminal Justice Treatment Practices (NCJTP) survey assesses practices in adult and juvenile correctional agencies in several areas: (1) client assessment of substance abuse and risk to public safety; (2) the nature, extent, and quality of substance abuse treatment services; (3) organizational factors that are likely to influence the implementation of evidence-based treatment practices; and (4) partnerships among and between criminal justice, drug treatment, and other agencies serving offenders. The survey is also designed to estimate national rates of treatment availability and access for offenders involved in different correctional and drug treatment programs and services. A unique feature is that the methodology includes a multilevel approach that captures the perspective of executives, frontline administrators, and line staff about current practices in a myriad of institutional and community correctional settings for adults and juveniles. The surveys were modeled, in part, after organizational surveys in the drug treatment delivery system (Chao et al., 2000; Lehman, Greener, & Simpson, 2002; Roman & Johnson, 2002), mental health systems (Drake et al., 2001), prevention programs in schools (Gottfredson & Gottfredson, 2002; Gottfredson, Najaka, & Wilson, 2001), and private industry (Baldauf, Reisinger, & Moncrief, 1999; Baruch, 1999; Tomaskovic-Devey, Leiter, & Thompson, 1994). This article is devoted to outlining the methodology of the NCJTP survey, including sampling design, survey administration methods and response rates, and human subject issues.

1.1. Lessons and methodology of NIDA’s three longitudinal surveys

The seminal DARP established a methodology that has been refined and improved upon in the past three decades. DARP, TOPS, and DATOS used nonexperimental, prospective, longitudinal designs to examine drug treatment effectiveness and service delivery. The emphasis in these studies was to obtain client level data in a national sample of community clinics and programs. Both client interviews and reviews of program records were conducted at intake, during treatment, and at set follow-up intervals. The most recent of the studies, DATOS, incorporated surveys of program administrators and counselors to gather data on program characteristics, treatment approaches, and procedures (e.g., intake, progress reports, etc.).¹ As noted by Fletcher, Tims, and Brown (1997) in their review of the contributions of this body of research, DARP established typologies for treatment programs, created a methodology for treatment research, and identified addiction outcome patterns related to readmissions, criminal behavior, and employment. DARP affirmed the importance of duration in treatment, with data that set the benchmark of ≥90 days to obtain positive benefits from treatment participation. TOPS extended previous findings and added the value of compulsory treatment on retention and the cost effectiveness of substance abuse treatment (Karberg & James, 2005). DATOS contributed knowledge on how treatment processes impact client outcomes (Simpson, 2004) and also included a survey of adolescent youth in drug treatment services, which was the first attempt to survey these specialized services (Grella, 2006). The NCJTP survey is designed to extend our understanding of the prevalence of treatment process practices in corrections-based programs and of factors related to their adoption and implementation.

1.2. Surveys in adult and juvenile justice settings

The Bureau of Justice Statistics [BJS] (1998, 2000), an arm of the U.S. Department of Justice, is largely responsible for conducting national surveys in adult corrections. The BJS has an active research portfolio, which consists of several ongoing surveys that gather data on adult populations under correctional control. As part of its portfolio, the BJS conducts a census of all jails to obtain and assess details on types of inmates, age and types of facilities, programs and health services, inmate employment, and costs. This census started in 1970 and has been conducted every 5 years. A similar survey is conducted of prisons on the same time frame, beginning in 1974 with the most recent census in 2000. The BJS assembled a census of state and local probation and parole agencies in 1979 and has modified it periodically. This survey is limited to probation populations, parole populations, or both, as legislatively defined, and does not include offenders on pretrial supervision, alternatives-to-incarceration or diversion programs, or offenders on other types of legal statuses in community settings (Anglin, Longshore, & Turner, 1999). The BJS developed a census of justice organizations that they use annually to estimate the size of the correctional population by setting (jail, prison, and probation/parole). The surveys are not designed to provide details on the services or programs offered to offenders, although special studies are undertaken periodically to address special topics. The Substance Abuse and Mental Health Services Administration (2006) conducted a mail and telephone survey in 1997 on the use of assessment tools to determine the need for substance abuse treatment services and the provision of substance abuse treatment services within federal, state, and local prisons and jails (Dillman, 1978). They did not include community correctional agencies.

The Office of Juvenile Justice and Delinquency Prevention (OJJDP), in collaboration with the U.S. Census Bureau, gathers national institutional corrections and court data on youth referred to the juvenile justice system. The most complete source of data on youth in corrections is the Census of Juveniles in Residential Placement (CJRP), conducted biennially since 1997. The CJRP provides 1-day population counts and basic demographic and delinquency case information. A second biennial OJJDP survey, the Juvenile Residential Facility Census (JRFC), focuses on operational data on facilities such as ownership, security, and crowding. Conducted on alternate years, the JRFC also gathers data on basic education, health care, and other service data (e.g., substance abuse assessment, drug–alcohol education, and group and individual counseling). Information on basic service provision is also included in a national survey of state correctional administrators conducted by the Council of Juvenile Corrections Administrators (CJCA); in the most recent 2005 survey, 46 jurisdictions reported data on agencywide services in education, substance abuse, mental health, and other areas (Ditton, 1999). As with the adult justice system, little systematic information is gathered on juvenile community corrections and primarily uses court dispositions data to infer community corrections data.

Although the above extant surveys capture data that highlight unmet needs of offenders, they provide an incomplete abstract collage that often generates more questions than answers. Major knowledge gaps exist in several areas: availability of various treatment modalities in different parts of the justice system (Anglin & Maugh, 1992; Belenko, 2002; Belenko & Peugh, 2005; Peters, May, & Kearns, 1992; Taylor, Fitzgerald, Hunt, Reardon, & Brownstein, 2001), the prevalence of evidence-based treatment practices in correctional settings (Farrington & Welsh, 2004; Landenberger & Lipsey, 2005; Lipsey & Landenberger, 2006; Lurigio, 2000; Mackenzie, 2000; Pearson & Lipton, 1999; Sherman et al., 1997; Simpson et al., 1999; Taxman, 1998a), and factors that foster or impede the implementation of effective treatment for substance-abusing offenders (Farabee, Prendergast, Cartier, Wexler, & Anglin, 1999; Taxman & Bouffard, 2000). The NCJTP survey was designed to gather baseline information on access, availability, utilization, and type and quality of extant treatment services in the juvenile system, criminal justice system, or both in various settings: institutional corrections (prisons and jails) and communities (probation, parole, and other venues).

2. The NCJTP survey

Unlike substance abuse treatment for the general population, where most programs are operated at the county or clinic level, the delivery of correctional programs and services involves a myriad of state, regional, and local organizations employing a mix of their own staff and contracted personnel, and services may involve multiple levels of government. Given the nature of justice settings, the project team identified the following goals for the survey: (1) to describe current drug treatment practices, policies, and delivery systems for offenders on probation or parole supervision, and in jails, prisons, and youth institutions; (2) to examine agency structures, resources, and other organizational factors that may affect service delivery, including mission, leadership, climate, culture, and beliefs about rehabilitation versus punishment; and (3) to assess coordination and integration across criminal justice agencies and between corrections and treatment systems. Tailored versions of the survey instrument were targeted to three sets of respondents operating agencies for the juvenile offender population, adult offender population, or both: (1) state executives administering correctional services; (2) administrators of institutional corrections facilities, community corrections offices, and community-based treatment programs; and (3) staff in select correctional and drug treatment facilities.

2.1. Sampling challenges and strategy

The goals of the survey presented challenges due to the lack of suitable and comparable sampling frames representing each segment of the adult corrections system or the juvenile corrections system. Because of the diversity of state and local criminal justice structures and the lack of independence among organizational units in these systems, it is difficult to identify complete and reliable frames from which to conduct census or probability samples. We addressed these challenges by employing a multiframe design consisting of a census of state executives responsible for correctional services and a national two-stage cluster sample representing individual correctional facilities and operational units (offices). To obtain the perspective of treatment directors and staff, the design also includes purposive samples of community-based treatment programs. The design and implementation of each of these samples are discussed below.

2.1.1. Census of state correctional agency executives and clinical coordinators, and state alcohol and drug abuse directors

The census of state executive directors of correctional agencies consists of executives responsible for prisons and community corrections (e.g., probation agencies, parole agencies, or both). In 24 states, one person is in charge of both adult institutional and community corrections; in the remaining 26 states, these agencies are independent of one another and headed by separate individuals. The names of directors were obtained by reviewing the Web site of each state and by working with the American Correctional Association (ACA), the American Probation and Parole Association (APPA), and the CJCA. In addition, with the help of the National Association of State Alcohol and Drug Abuse Directors (NASADAD), we identified and surveyed all directors of alcohol and drug abuse directors in each of the 50 states responsible for adult and juvenile services.

A second part of the census targeted the administrator in each state criminal justice agency who was responsible for overseeing substance abuse treatment (and often other correctional services) in the agency. Although it had been anticipated that each state criminal justice agency would have one clinical coordinator, initial survey mailing found that state correctional agencies did not frequently have a single person responsible for and knowledgeable about treatment programs and services. This required us to solicit surveys from multiple respondents in the agency and then aggregate data for a single-state agency-level response.

2.1.2. Adult prison sample

The most recent available census of prison facilities in the United States was conducted by the BJS in 2000 (Stephan & Karberg, 2003) and served as a frame for this study. The census consisted of 1,668 private, state, and federal correctional facilities. Of these, 1,317 were state prisons, 3 were District of Columbia facilities, 84 were federal facilities, and 264 were private facilities. We first excluded the federal prisons from our adult prison sample frame because these represented a unique system serving a relatively small number of prison inmates nationally (in 2005, federal prisoners made up 12% of the total prisoner population in the United States; Harrison & Beck, 2006; Mumola, 1999). Also excluded were institutions identified by the BJS as community corrections facilities (426); specialized prisons responsible for reception, diagnosis, or classification facilities (50); short-term hospitals and other medical facilities (9); and prisons exclusively devoted to youthful offenders (19), chronically mentally ill inmates (12), and geriatric care (3). After closely examining the 426 "community corrections" facilities to verify their purpose, function, and size, it was decided to include them in the community sampling frame (see below). The final sample frame consisted of 1,065 facilities, including 938 general confinement prisons, 58 facilities specializing in alcohol/drug treatment, 45 boot camps, 14 institutions for returned-to-custody inmates (parole violators), and 10 others.

Budget considerations dictated a target sample of 150 adult prison facilities. Given the study goals, the 58 facilities identified in the BJS frame as focusing on alcohol treatment, drug treatment, or both were sampled with certainty. In selecting the remaining facilities, we followed the same strategy employed by the BJS in their national prison studies. The BJS’ stratified sampling frame breaks the country into eight regional categories, including four states with the largest correctional populations and the remaining states grouped into south, west, midwest, and northeast regions (Harlow, 2003). States and regions are not identified here to help protect the confidentiality of responses. Within these strata, the 92 remaining prisons were selected randomly, in probabilities proportional to the size (PPS) of the facility. The facility’s population as of June 30, 2000, served as the measure of size (MOS). When used with stratification, sample selection by PPS without replacement is the most appropriate procedure when survey units vary in size and are unequally distributed in the population (Kish, 1965). Our PPS selections were implemented with SAS/STAT (PROC SURVEYSELECT) software version 8.1 of the SAS System for Windows (SAS Institute, 2000).

2.1.3. Juvenile residential facilities sample

The sampling strategy for juvenile residential facilities employed the same logic as the adult prison sample in a sampling frame constructed from the (ACA) 2003 Directory of Adult and Juvenile Correctional Departments, Institutions, Agencies, and Probation and Parole Authorities. Issued annually, the directory is based on responses to ACA questionnaires and self-reports by state corrections agencies in all 50 states and a few large city and county agencies. If an annual update is not submitted by a facility, the information from the prior edition is reprinted. The 2003 ACA directory listed 1,017 juvenile institutions and community corrections facilities. Nineteen institutions that were in the 2000 BJS census of adult prisons and that served youthful offenders exclusively were added to the initial sample frame, resulting in 1,036 juvenile facilities and offices. The ACA directory frame was chosen because the other most comparable publicly posted database, the federal juveniles in placement databook, was dated from the mid-1990s.

As with the adult prison sample, several eligibility exclusion criteria were applied. First, 299 facilities that were described in the directory as community corrections facilities, diagnostic and reception centers, group homes, and other specialized facilities (designated for runaways, foster care youth, etc.) were excluded. Second, because our intent was to examine services in large state-funded secure institutions used primarily for committed youth, we excluded 315 facilities with capacities of fewer than 25 persons. Both community corrections facilities and residential facilities and detention centers holding <25 youth were considered as part of the community frame. Phone contacts and Web searches of the remaining 422 facilities revealed that a few facilities had closed or merged with other facilities, and these were also excluded from the final list. The final frame used for sampling juvenile residential institutions totaled 408 facilities, of which approximately 23% were privately run.

From this frame, a target sample of 72 juvenile residential facilities were selected using PPS (without replacement) methods, stratified by the same eight-category region variable used in the adult prison sample, and the average daily population (ADP) of the facility. The MOS was defined as the facility capacity or ADP from the ACA directory, updated in some cases from information gathered in phone calls or Web searches. Because MOS varied greatly across facilities and regions, maximum MOS caps were set independently for each of the regions.

2.1.4. Community sample

The delivery of community-based justice, drug treatment services, or both varies considerably across the United States and within states. The survey included all facets of the community-based system, including jails and detention centers, probation, parole, local community correctional agencies, and other community corrections agencies and diversion programs. The BJS has a sample of adult probation and parole agencies that was constructed in 1991, but after consultation with the BJS about the adequacy of this frame and consideration of recent changes in the community corrections field (e.g., the expansion of intermediate sentencing programs, passage of the Community Corrections Act, growth of reentry initiatives, moves to privatize many supervision services, and redistribution of supervision among state and local agencies, etc.), it was determined that this frame was outdated and incomplete (A. Beck, personal communication, 2004). Because no comprehensive list or directory of local correctional agencies and programs exists, we elected to employ a standard two-stage stratified cluster sampling strategy (Kish, 1965). The first-stage cluster was composed of counties or county equivalents as defined by the U.S. Census Bureau and the National Institute of Standards and Technology (see http://geonames.usgs.gov/fips55.html), and the second stage was composed of corrections facilities, offices, or programs within selected counties.²

The first stage involved selecting a sample of 72 counties from among 3,141 counties or county equivalents identified in the 2000 Census. Because counties vary greatly and are highly skewed in population size, we again employed PPS without replacement methods to select the first stage of the sample. The MOS was the total county population according to the 2000 Census, using a three-category population size variable (small = <250,000; medium = 250,000–750,000; large = >750,000). We formed 24 strata by cross-classifying the size variable with the same 8-category BJS region variable. Counties with >3 million persons were selected into the sample with certainty. This method assured balance in resulting PPS selections.

The second stage involved a purposive sample of programs, facilities, and organizations. Each of the 72 counties was investigated via the Web, direct phone calls to agencies and facilities, and other sources (e.g., mailing lists of national associations, informer networks, etc.) to identify targeted recipients of the local community survey. The lists of community correctional programs obtained in drawing the adult and juvenile facility samples were also cross-referenced to identify units operating in these counties. We assembled a list of survey sites within each county and then recontacted one or two key informants in the county (such as a probation chief or a coordinator) to confirm the local array of correctional facilities, offices, and programs. This process yielded 644 targeted respondents for the 72 counties that represented both adult and juvenile jails and detention centers (local or regional); community correctional facilities; probation offices, parole offices, or both; and specialized correctional programs (e.g., TASC, alternatives-to-incarceration programs).

To provide a more complete picture of local substance abuse service delivery for offenders, we also added a purposive sample of community-based treatment programs that serve criminal justice clients drawn from the National Survey of Substance Abuse Treatment Services (Office of Applied Statistics, 2003). Due to resource constraints, in each of the 72 sample counties, we targeted up to 5 programs with the largest adult client capacity and up to 3 programs with the largest adolescent (<18 years) client capacity (some counties had fewer than the target number of programs). The total target sample for the community treatment programs was 243.

3. Procedures

3.1. Survey respondents

The survey respondents were executives of state agencies, executives in correctional agencies responsible for programs and services, or administrators responsible for the facility, office, or program identified in adult prison, juvenile residential facilities, and community samples. The latter includes prison wardens, directors of juvenile facilities, jail wardens and directors (who were sheriffs in some counties), and administrators responsible for local probation and parole offices. If a sampled correctional institution or office operated a substance abuse program of any kind (based on information from Web searches, state agency reports, and phone calls), the director of the program was also surveyed in a manner similar to that of the directors of community-based drug-free outpatient facilities selected in the community sample. The names, titles, and addresses of respondents were obtained from directory lists, Web searches, and phone calls.

We also sought the viewpoint of line staff in these facilities and, thus, targeted a convenience sample of line personnel from a subset of the correctional facilities and treatment programs identified for administrator surveys. Resources limited this part of the survey to staff working in prison and community-based agencies and programs that were located in states covered by the CJ-DATS research network. In 11 of the covered states, the staff-level convenience sample was limited to 1 prison and 1 county; in 3 very populous CJ-DATS states, 2 counties and 2 prisons were included in the sample. If a correctional agency or director in a targeted facility refused to allow staff to participate in the study, another facility from the administrator sample was substituted. For the staff survey, each facility provided a list of employees by job category, and researchers randomly selected the staff to be surveyed. In some select areas, a liaison at the sample agency handed out survey instruments to staff.

3.2. Instrumentation

Surveys tailored to the different respondent groups were developed by a team of investigators from the CJ-DATS research collaborative. The team consisted of 15 scientists, with a work group of practitioners. Senior researchers on the team all had a minimum of 20 years of work in the fields of substance abuse, criminal justice, or both. Each center from the CJ-DATS cooperative had a senior scientist participating in the process of developing the instruments, and, collectively, the group had experience in conducting research in substance abuse treatment, jails, probation and parole, therapeutic communities, survey methodology, and organizational behavior. The team engaged in a development process of nearly 1 year to develop and pretest the survey instruments.

Researchers used preexisting scales for most of the domains, particularly in organizational assessment areas, where there are numerous measures with extensive prior use and proven psychometric properties. Table 1 provides a list of the scales used in the survey version of each respondent group.³ New scales were created in three areas: attitudes toward punishment and rehabilitation, systems integration, and knowledge of effective practices in drug treatment and corrections. A manual of the scales and their psychometric properties is available from the senior author upon request. The instrument also consisted of the descriptive characteristics of the correctional agency that were adapted from the BJS and OJJDP surveys.

Table 1.

Instrumentation for the survey

Type of items	State CJ executives	State CJ clinical coordinators	State AOD agency executives	Facility administrators (prison, probation, parole, etc.)	Staff
Respondent characteristics	X	X	X	X	X
Organizational characteristics	X	X	X	X	X
Correctional programs (e.g., size, nature, etc.) characteristics	X	X	X	X
Substance abuse treatment programs characteristics	X	X	X	X
Social networks/agencies collaborate	X	X	X	X
Integration of services with other agencies (Taxman & Young, 2004)	X	X	X	X
Attitudes toward punishment and rehabilitation (personal values; Young & Taxman, 2004)	X	X	X	X	X
Organizational needs assessment (Lehman et al., 2002)	X	X	X	X	X
Organizational climate (Orthner, Cook, Sabah, & Rosenfeld, 2004; Scott & Bruce, 1994)	X	X	X	X	X
Organizational culture (Cameron & Quinn, 1999; Denison & Mishra, 1995)	X	X	X	X	X
Climate for treatment (Schneider, White, & Paul, 1998)		X	X	X	X
Cynicism toward change (Tesluk, Farr, Mathieu, & Vance, 1995)		X	X	X	X
Organizational commitment to treatment		X	X	X	X
Leadership (Arnold, Arad, Rhoades, & Drasgow, personal communication, 2000; Podsakoff, MacKenzie, & Fetter, 1990)				X	X
Treatment practices beliefs (Melnick & DeLeon, 1999; Melnick, Hawke, & Wexler, 2004; Young & Taxman, 2004)		X		X	X
Perspective taking (Parker & Axtell, 2001)				X	X
Intradepartmental coordination (Georgopoulos & Mann, 1962)				X	X

Extensive reviews and pretesting helped shape the final versions of the surveys. Draft instruments were reviewed by staff of national associations representing corrections policymakers and practitioners, including the ACA, the APPA, the National Treatment Accountability for Safer Communities, the National Association of Drug Court Professionals, and the NASADAD. One version of the instrument was pretested using a focus group of senior staff of the ACA. Pretest versions of the prison administrator and staff surveys were administered to 31 wardens and nearly 1,000 corrections and treatment staff in one state correctional system. The community survey was pretested in several correctional and drug treatment facilities in one county, which consisted of a mix of state and local justice agencies. This pretesting was conducted to determine whether the psychometric properties underscoring the scales were applicable in a community correctional setting, to identify wording issues, and to work on survey procedures. The convenience-based pilot state and county agencies were not intended to be nationally representative, but provided an opportunity to test the feasibility of the study procedures and instrumentation as designed by the research team.

3.3. Survey administration, follow-up procedures, and response analyses

The mode of the survey was a self-administered paper and pencil questionnaire. The survey of state executives consisted of 14 pages; the state clinical coordinator survey consisted of 35 pages; the facility, program, and office administrator survey consisted of 27 pages; and the staff survey consisted of 22 pages. The length of the survey was dictated by the content, with longer surveys required to generate estimates of program availability, utilization, and capacity, and to provide contextual information on organizational factors and policy and program elements. The pretest survey took approximately 60–90 minutes to complete; respondents reported a similar time consumed to complete the survey.

Organizational surveys tend to have response rates lower than those of surveys of general populations (Baldauf et al., 1999; Baruch, 1999; Tomaskovic-Devey et al., 1994). Response rates to surveys of executives and other senior administrators hover around 20–30%, whereas surveys of employees and midlevel managers report somewhat higher response rates (Baruch, 1999). Response rates in organizational surveys, such as those of general populations, have declined in time, as the response rate to such surveys was 64% in 1975 but had dropped to 48% by 1995 (Baruch, 1999). A similar pattern is reported for surveys of military personnel (Newell, Rosenfeld, Harris, & Hindelang, 2004). Mailed self-administered questionnaires result in rates of response lower than those of other modes, yet mailing is the least expensive and most often employed survey means (Hager, Wilson, Pollak, & Rooney, 2003). As shown in Table 2 below, our response rate varied with 71% of state executives, 72% of clinical coordinators, 63% of facility administrators, 51% of local treatment directors, and 34% of staff responding to the survey.

Table 2.

Response rates for each survey level and type of respondent

	Adult		Juvenile		Total
Survey level	Final N	Response rate (%)	Final N	Response rate (%)	Final N	Response rate (%)
S1: Survey of executives
Total	100	74.6	70	66.7	170	70.8
S2: Survey of corrections clinical directors and alcohol and drug agency directors
Total	98	72.1	70	70.7	168	71.5
Corrections clinical directors	59	69.4	41	66.1	100	68
Alcohol and drug agency directors	39	76.5	29	78.4	68	77.3
S3: Survey of administrators and treatment directors
Total	431	67.4	216	54.4	647	62.5
Prisons/juvenile detention centers	98	69.5	49	73.1	147	70.7
Local corrections (jails, probation, parole, and other community corrections)	191	71.5	92	56.4	283	62.1
Treatment directors in prisons/juvenile detention centers	66	61.1	29	55.8	95	59.7
Treatment directors in local community-based treatment facilities	76	61.8	46	39.7	122	51
S4: Survey of staff
Total	734	37.2	351	28.5	1,085	33.9
Prison	178	30.2	73	25.9	251	28.8
Community corrections/jail	422	41.3	190	28.1	612	36.1
Treatment	134	36.9	88	32.2	222	34.9

Numerous efforts were made in all areas of the survey (from design to response analysis) to address issues of response rates and potential for response bias.⁴ First, researchers developed a strategy that incorporated the national criminal justice and drug treatment professional associations as partners in the survey administration. The cover letter for surveys was tailored to the audience: The instrument sent to correctional agencies was authored by the executive director of the ACA, and the letter to drug treatment agencies was penned by the executive director of the NASADAD. Accompanying letters and notifications about the survey also came from executives of the APPA, the CJCA, the American Jail Association, and the National Treatment Accountability for Safer Communities. Second, we employed the Dillman (2000) method of follow-up procedures, which included reminder postcards, letters from national associations, frequent reminder phone calls, faxes, e-mails, replacement surveys, and personal contact by the research collaborative team.

During the data collection period, a series of analyses was conducted to assess whether organizational factors were associated with response patterns. Response bias is typically measured as the product of two quantities: the proportion of the sample failing to respond and the difference, if any, in the mean value of an outcome variable observed among responding units (Groves, 1989). The response rates by themselves are not sufficient to indicate whether response bias is important enough for concern. We fit a negative binomial (Poisson with over-dispersion) regression model for the number of contacts needed to return a completed interview. Independent variables included several organizational indicator scales, including organizational climate, culture, resource needs, and supervisor’s leadership style. Details regarding the methods used in this analysis are described elsewhere (Wiersema & Taxman, 2006). We found that, controlling for region and facility population, no organizational factor was a significant predictor of the number of contacts needed to produce a response. Based on these results, we assume that unit nonresponse can be treated as "missing at random" (Little & Rubin, 1987).

Survey procedures also included data editing and verification. Upon receipt, surveys were reviewed for completeness. If part of the survey was incomplete, calls were made to the survey respondents to inquire as to whether they would complete the remaining portions over the phone. Most of the respondents completed the survey. In addition, data verification techniques were employed to address information that was incomplete or inconsistent regarding the number of offenders under correctional control, size of programs/services, or nature of the programs/services offered. The researchers also verified key information in the survey instrument (e.g., size of correctional population, annual intake numbers, and size of the largest correctional or substance abuse programs) through phone calls to a random sample (20%) of respondents to the surveys of correctional clinical coordinators and administrators.

3.4. Human subjects procedures

The survey procedures were reviewed and approved by the Institutional Review Board (IRB) overseeing research at each of the 11 research centers comprising NIDA’s CJ-DATS network. The study is considered of minimal risk because questions mainly pertain to work related issues, although there are personal values and opinion questions regarding program and agency operations. It had been anticipated that local administrators and staff would need permission from their agency director to participate in the study during work time, so a generic permission letter addressed to agency personnel was included in the survey package sent to the agency executive, along with a letter explaining the study and requesting that they sign and return the permission letter for inclusion in subsequent mailings to agency administrators and staff. This package with the letters and the executive survey was sent out to other levels of the organization about 3 months before the surveys.

During the course of survey administration, several state correctional agencies contacted the research team to indicate that they had their own IRB procedures. The research team prepared special IRB packages for four adult correctional and two juvenile agencies. These approval processes varied from short informal letters requesting approval to formal full-length IRB applications. All agencies gave permission for the survey of administrators, but one agency did not allow its correctional and treatment staff to complete the survey.

4. Response rates and sample weighting procedures

4.1. Adult prisons

Of the 150 included in the final sample of prisons, 141 were determined to be eligible for the survey (8 of the facilities closed and 1 was no longer operating as an adult prison). Surveys were returned by 98 of 141 eligible prisons, for a response rate of 69.5%. Although response varied across regional strata (54–85%), a logistic regression modeling probability of response given region and facility size did not show any statistically distinguishable differences. This, together with the contact effort analysis discussed above, lends additional confidence to the assumption that nonresponding prisons are missing at random (i.e., their failure to respond was not due to some systematic process that might influence observed responses to principal outcome variables).

Given this assumption, we adjusted sampling weights (w_i = the inverse of the probability of selection) with a standard sample-based procedure that distributes the weights of nonresponding units (i ∈ N) to those that responded (i ∈ R)—a standard procedure (Elliot, 1991). The adjustment factor is:

$$A^{(\text{nr})}=\frac{{\Sigma }_{i\in R}{w}_i+{\Sigma }_{i\in N}{w}_i}{{\Sigma }_{i\in R}{w}_i}.$$

This adjustment was computed for each sampling stratum and multiplied by the base sampling weight for each individual prison to produce the prison’s adjusted weight:

$$w_i^{(\text{nr})}=w_i{A}^{(\text{nr})}.$$

A final revision of this nonresponse-adjusted weight involved examining the distribution of the weights and trimming extreme values to no more than 3 SD of their mean. Trimming is a standard practice in survey methodology because extreme weights can unduly inflate the variance of survey estimates. Most statisticians believe, however, that the reduction in variance decreases mean squared error and is thus an acceptable tradeoff of a potential bias. We used the inspection method described in Potter (1988, 1990) to choose a conservative trimming point for the adult prison sample. Applying this weighting procedure to the ADP value reported by adult prison respondents in the NCJTP sample and aggregating across the responses yields a point estimate of 1,233,867 for the adult prison population ADP in January 2005—a figure 0.9% lower than the BJS’ 2004 estimate of 1,244,867 (Table 1 of Harrison & Beck, 2005) (Table 3).

Table 3.

ADP in the criminal and juvenile justice systems: Comparison of NCJTP point prevalence estimates with other existing studies

Variable	Estimated number of offenders under correctional control (NCJTP survey)	Size of correctional population from existing studies	Percentage difference
Number of adults in prison	1,233,867	1,244,867a	−0.09
Number of adults detained in jails	745,766	713,990a	4.5
Number of adults on parole	908,477	675,534b	34.5
Number of adults on probation (Mumola, 1998)	3,949,089	4,122,779b	−4.4
Number of adults on diversion, alternative, or other supervision status	1,006,586	NA	NA
Number of youth committed to residential facilities	57,355	58,818c 54,208d	−2.5 5.8
Number of youth on probation/parole	542,349	NA	NA
Number of youth on other forms of supervision	20,620	NA	NA

Note. The table excludes persons under federal jurisdiction.

^aFrom Harrison and Beck (2005).

^bFrom Glaze and Palla (2005).

^cFrom the American Correctional Association (2003). The figure is adjusted for youth in community corrections facilities, diagnostic centers, and group and foster homes.

^dFrom Snyder and Sickmund (2006). The figure is adjusted for youth in diagnostic centers and group homes, and facilities with <25 youth.

4.2. Juvenile residential facilities

The final survey of juvenile residential facilities consisted of 67 facilities (5 of 72 facilities in the original sample were determined to be operating as both adult and juvenile facilities and were considered ineligible). Of the 67 sampled facilities, 49 responded to the survey, for a response rate of 73%. The response varied by region (50–100%). Similar to the adult prison response analysis, we fit a logistic regression model predicting response propensity. In this analysis, facility size did not predict response propensity, although one region was found to be significantly less likely to respond than those in other regions. This is not cause for concern, however, because the sample size is small (i.e., half the size of the adult prison sample) and because we found no evidence of response bias in our contact effort analyses (Wiersema & Taxman, 2006). Given this, we performed a missing-at-random nonresponse adjustment to the sampling weights, as used for the adult prison sample. Due to the smaller samples and lower response rates in the juvenile sample, we modified the trimming procedure so extreme values were no more than 2 SD of their mean. This reduced the possible biasing effects of two very low probability counties included in the final community sample. The juvenile residential facilities population estimate from weighted NCJTP responses was very similar to the census reported in the ACA (2003) directory when adjusted for nonsampled reception centers and group and foster homes. The NCJTP estimate also corresponds with the figure reported by the OJJDP for the number of youth in residential placement in 2003 (Snyder & Sickmund, 2006) when the latter is adjusted for reception centers, shelter and group homes, and facilities housing fewer than 25 youth (all of which were excluded from the NCJTP sampling frame).

4.3. Local corrections and treatment programs

One thousand forty-six respondent facilities, offices, and programs were identified as our target sample for both juvenile and adult community-level surveys. Of these, 647 responded, for a response rate of 61.9%. (The response rate for adult units was 66.6%, and the rate for juvenile units was 54.1%.) Response varied by region from 46% to 80%. These response rates were above the norm for organizational surveys. Survey editing procedures validated that each of the response units was independent of other responders in a given organization.

Given the two-stage cluster sampling strategy, statistical adjustments for nonresponse are not straightforward. Although our second-stage selections are purposive (i.e., not a probability sample), the first-stage sample of counties is based on known probabilities of selection. To the extent that a nonresponding organization is the only one of its type within a county, we can apply a missing-at-random nonresponse adjustment to its sampling weight. However, if there is more than one organization of that type within a county, it is not clear what the appropriate adjustment should be if one or more of the organizations failed to respond. Given these complexities, we elected not to adjust the sample for nonresponse. The use of first-stage weights to aggregate responding second-stage units to the national level should result in conservative population estimates, which appear appropriate given the nature of this survey.

The purposive nature of the treatment agency’s second-stage selections should also produce population estimates that likely understate their "true" values because only some units within a county were surveyed. These estimates may be biased in the sense that the study only includes the largest of these operations in the jurisdiction that serves criminal justice or juvenile justice populations. Nonetheless, an argument for their utility can be made in that they serve as useful lower bounds for areas of research that have few, if any, benchmarks for comparison. Future research will, no doubt, build on and refine our sampling strategy to produce better estimates.

4.4. Correctional and line staff

As anticipated, the survey of correctional and line staff was the most difficult component of the survey. Most of the responsibility for this part of the survey was placed on the CJ-DATS collaborative research centers to take advantage of any established working relationship between the center and the correctional and treatment agencies within their research catchment area. Overall, 56% of all selected respondent sites participated in this part of the survey, ranging from 65% of adult prisons to 42% of juvenile residential facilities. The reasons cited by facilities for nonparticipation included: unwillingness to allow the use of work time to complete surveys, administrative restrictions, length of the survey instrument, and nonresponse to phone calls, e-mails, or both. Each research center used its own procedures for working with selected agencies; some took survey packets to the sites and brought lunch or other snacks to give out to staff, whereas others obtained staff lists from agency liaisons and sent survey packets (with a consent form, survey, and stamped return envelopes) directly to these individuals for dissemination.

Sixty-eight sites sent lists of employees. An additional 49 sites provided the number of surveys needed. Three thousand three hundred seventy-five surveys were sent out to all sites between February and November 2005. Due to IRB requirements, one of the research centers did not distribute surveys by site. (Instead, it held a luncheon for all sites and distributed the surveys at that time.) Follow-up was done by the research center either in-person or through phone calls or e-mails to site liaisons. One thousand one hundred seventy-two staff surveys were returned, for a response rate of 34.7%. The response rates for individual sites ranged from 0% to 100%. Sites where research staff went in-person to distribute surveys and to conduct follow-up had the highest response rates, including one CJ-DATS center that had an average response rate of close to 60%.

4.5. Estimated adult and juvenile correctional populations

Using the above techniques to obtain point estimates, Table 3 presents the results of applying weighting procedures and comparisons to existing published studies, where they exist. As suggested in our discussion of the methodology, the community sample resulted in estimates of the formal probation and parole populations and those offenders in other "correctional statuses" that are not typically found in other surveys (e.g., diverted programs, alternatives-to-incarceration programs, community correctional programs, etc.), and because no estimates are available for juveniles under formal community supervision, this survey provides the first estimates of this population.

For the most part, the NCJPT survey provides similar point estimates of adults incarcerated in prisons and juvenile offenders in residential facilities. The community sample illustrates the diversity of the types of populations (e.g., probation, parole, community corrections, etc.). Because different jurisdictions use different terms to describe statuses of offenders (e.g., some jurisdictions refer to offenders released from jail as parolees, others use probationers, etc.), the differences with the other cited studies were expected. The findings are that approximately 4.8 million adults are under formal probation supervision, parole supervision, or both; slightly >1,000,000 adults are under the control of community correctional agencies; about 540,000 youth are under formal juvenile supervision; and just >20,000 youth are in diversion or alternative programs.

5. Discussion and conclusions

The NCJTP survey provides the first survey of the correctional landscape, across both adult and juvenile justice systems, to describe and assess substance abuse treatment programs and services provided to offenders. As will be discussed in the following articles, the information from the survey provides an unusually textured view of correctional substance abuse service delivery and should be useful in better understanding offender outcomes from these services. From a methodological point, this survey has contributed significantly to knowledge on the conduct of surveys among correctional agencies existing across the correctional landscape and our understanding of challenges to ensuring the accuracy of survey data.

Overall, point population estimates for the survey mirror other existing sources and indicate the robustness of the methodological approach. The methodology of using a census of state administrators and a two-stage multiple samples of communities was successful. The two-stage approach ensured that the sample is representative of national trends and allowed the researchers to develop a frame for the second stage, which reveals the range of agencies involved in service provision at the local level. In analyses presented elsewhere in this volume, the survey revealed 24 combinations of state, local, and nongovernmental agencies that operate to provide correctional services to youth and adults in the 72 sampled counties. In most counties, the organizational structure for youth is different from that for adults. Part of the goal of this survey was to learn about how programs and services are provided, which was accomplished. The survey methodology also provides another method of administering surveys to what are typically government agencies.

From a survey administration perspective, the survey’s response rates were likely a function of the tenacity of the research team and the involvement of key stakeholder associations that assisted the process. This survey was truly a collaborative effort, involving national organizations that took part in various phases of survey development and implementation. From the earliest planning discussions, the research team was committed to involving field leaders and representatives, and the better-than-expected response rates are a product of these partnerships. The survey cover letter, accompanying letters of support, and follow-up letters from national associations were used to illustrate the importance of the survey findings to the field. The letters also addressed the policymakers’ demands for information on service structures and practices. This approach echoed the guiding tenets of CJ-DATS, which emphasize partnerships between researchers and practitioners, and the practical application of research findings. We believe that the cover letters on the surveys authored by national associations added to the impressive response rates; in fact, when some respondents lost the accompanying envelope, they would forward the survey to the association, and in some follow-up phone calls, the respondents referred to the survey as coming from a given association.

Conducting a survey such as this one has provided numerous lessons. First, some of the lower response rates are likely due to the length of the survey instrument. This survey was one that tried to meet a number of objectives and, in doing so, is long (three of four had >20 pages). By creating sections to the survey, which allowed the user to have an easy "rest point," we believe that we were able to get responses that did not speak to fatigue. During the survey administration, we learned that nearly 20% of the facilities had acting administrators. Some of the acting administrators were more willing than others to complete the survey or were hesitant to complete the instrument either due to their "unofficial" capacity or due to the concern that the survey will detail how little services are available in their facility. These are challenges that future researchers will have to address. In retrospect, using replacement strategies and a slightly larger budget to enlarge the sample size might have been preferred, but due to the study’s limitations, we chose to put funding into proactive, engagement, and follow-up strategies, sending out replacement surveys instead of replacing sites. Finally, another lesson regards the need to clarify more about the nature of the populations under the jurisdiction of the agency. Although we relied upon the categories in the field, more attention to the different components of the "legal status" of the offender is probably worthy of further consideration.

Throughout the process of planning, conducting, and presenting survey results, we have been confronted with difficulties that stem from the inadequate and imprecise language used in the field of corrections to describe services, programs, and systems. Just those three words—services, programs, and systems—have different meanings to different individuals and are used inconsistently. One drawback of imprecise terminology is that administrators may overlook or misunderstand critical information because it appears to them to be irrelevant to their own system. A case in point is jails that operate as both secure institutions and reporting centers or programs for offenders residing at home or in nonsecure facilities, or community correctional programs that operate halfback programs where offenders are in a semi-incarcerated status. Developing consensus on an improved precise taxonomy or vernacular for use by the field is an important step in establishing a strong knowledge base.

In the field of substance abuse treatment, particularly corrections, the survey is unusual in its focus on organizational issues. In this regard, the survey reflects a growing concern about the role played by organizational factors in the gap between science and practice. Reviews of literature point to principles of effective treatment, such as matching offenders to treatment modalities based on need (and not criminal charge) and providing continuity of care from institutions to communities and residential-to-outpatient programming that have longstanding support in research but remain unimplemented (Andrews & Bonta, 1996; CSAT, 1998; Lowenkamp, Latessa, & Hoslinger, 2006; NIDA, 2006; Simpson et al., 1999; Taxman, 1998a, 1998b; Taxman & Bouffard, 2000; Taxman & Marlowe, 2006). The survey was designed to add new insights and to form new strategies that address these age-old issues. Overall, we believe that the survey findings should also help address gaps inherent to an existing body of literature that tends to rely upon site-specific studies conducted in jurisdictions that are known to pilot innovations (e.g., Maricopa County, AZ), have the capacity to support extensive research-driven training of program staff (e.g., the state of Delaware), or have unusually strong working relationships with a researcher. The survey is a first step in beginning the process of knowledge dissemination and diffusion to improve practice (Institute of Medicine, 1992).