Development and initial reliability and validity of four self-report measures used in research on interactions between police officers and individuals with mental illnesses

Abstract

There currently exists a dearth of reliable and valid instruments to examine key police officer variables of importance in the growing research on their interactions with individuals with mental illnesses. This study tested reliability and validity of four newly designed measures of the constructs of self-efficacy (Self-Efficacy Scale; SES), referral decisions and de-escalation skills (Behavioral Outcomes Scale; BOS), attitudes toward psychiatric treatment (Opinions about Psychiatric Treatment; OPT), and social distance (Adapted Social Distance Scale; ASDS) in a sample of law enforcement officers. Self-administered, anonymous surveys, which included the measures of interest, were completed by 177 officers—68 of whom were undergoing Crisis Intervention Team (CIT) training and 109 of whom were not—at the beginning and end of week-long trainings. Analyses examined the internal consistency reliability, test-retest reliability, and construct validity of the instruments. The four measures of interest were found to be reliable and valid. Specifically, internal consistency coefficients and test-retest reliability correlations were generally acceptable, all four demonstrated sensitivity to change, and validity correlations were significant and in the expected direction. Findings demonstrated the ability to measure key constructs related to attitudes and intended behaviors in law enforcement officers utilizing psychometrically sound instruments. Further testing and the development of additional reliable and valid instruments focused on attitudinal and behavioral domains among officers who have frequent interactions with individuals with mental illnesses would be of great value.

1. Introduction

Deinstitutionalization has partly evolved into a trans-institutionalization of people with mental illnesses. Since state mental facilities began efforts to transition patients to community-based alternatives in the 1960s, the criminal justice system has experienced a noticeable increase in the presence of persons with mental illnesses among the jail and prison populations (Lamb and Bachrach, 2001). According to the U.S. Department of Justice, approximately 16% of all inmates in state adult correctional facilities and city/county jails have a mental illness (Ditton, 1999). Police officers across the country have been placed in a position to determine the disposition of persons with mental illnesses on a daily basis; opting between referrals to mental health services and jail (Council of State Governments, 2002). Some have estimated that approximately 7% of police encounters involve a person with a mental illness in departments of cities with a population >100,000 (Georgia Association Chiefs of Police, 2008). Thus, the interactions between police officers and persons with mental illnesses represent a crucial, but grossly understudied area of mental health services research.

The surge in incarceration has heightened awareness of the impact of interactions between police officers and persons with mental illnesses. Those from the criminal justice system, mental health professions, and consumer networks have recognized this issue, and through collaboration of these stakeholders and other diverse stakeholders, the Crisis Intervention Team (CIT) model, a pre-booking jail diversion program has been implemented and broadly disseminated across the United States and internationally (Dupont et al., 2007). The CIT model consists of multiple elements (ongoing, operational, and sustaining) to ensure successful implementation, effective training, and long-term sustainability. One component of the model that has been adapted by many police departments is the voluntary 40-hour training for law enforcement officers, which consists of didactic lectures, site visits to mental health facilities, and de-escalation training through role-play activities (Dupont et al., 2007).

Given the noteworthy prospect of the Crisis Intervention Team (CIT) model of collaboration between law enforcement, advocacy, and mental health professionals, and its widespread and expanding implementation in communities nationwide, further research is needed to examine the effectiveness of CIT for improving outcomes in officers’ interactions with individuals with mental illnesses and to measure an array of officer-level variables (Compton et al., 2008). However, there is a substantial lack of instruments that have been validated in officer samples. Crucial factors to assess include improvement in self-efficacy, appropriate use of verbal de-escalation skills, appropriateness of referral decisions, enhancement of attitudes toward psychiatric treatment, and reductions in social distance and other forms of stigma, each of which is very briefly reviewed below.

Self-efficacy, as defined by Bandura (1977; 1986), is a person’s belief in his or her capabilities to successfully perform tasks in particular occurrences that affect their lives. Thus, self-efficacy can be conceived of as circumstantial confidence; individuals with high assurance in their capabilities can overcome difficult tasks and cope with these challenges to satisfy specific situational demands (Bandura, 1982). Furthermore, an individual’s self-efficacy belief can determine how one feels, thinks, and behaves (Bandura, 1977). In terms of responding to mental health-related calls, CIT training should improve officers’ self-efficacy, translating into constructive interactions when encountering those with a mental illness. Bahora and colleagues found that CIT-trained officers reported enhanced self-efficacy when responding to vignettes portraying individuals with schizophrenia, depression, alcohol dependence, and cocaine dependence (Compton et al., 2008). This improvement in self-efficacy likely leads to changes in behaviors, as proposed by Bandura (1977). In the CIT model, key behaviors of significance are de-escalation skills and referral decisions, both of which must be reliably and validly assessed.

As CIT officers are trained to serve as front-line responders to mental health crises (and when suitable, re-direct individuals with mental illnesses to treatment services, instead of the legal system) (Dupont et al., 2007), officer’s attitudes concerning psychiatric treatment are critical to the successful delivery of proper care to individuals with mental illnesses. In addition to concerns about such attitudes related to psychiatric treatment, desired social distance, a form of stigma, also needs to be evaluated. Social distance can be described as significant physical, social, or psychological distance, or reduced closeness, that characterizes an interpersonal relationship (Park, 1924). In relation to CIT-trained officers, measuring social distance stigma seems quite salient. Reductions in desired social distance can be hypothesized to lead to better interactions with those with a mental illness through an increased level of comfort; better rapport and communication between the officer, the individual, and his or her family; as well as improved safety for all parties. Therefore, better outcomes can be expected for individuals in crisis (Bahora et al., 2008; Compton et al., 2006).

Though vital to a better understanding of the impact of CIT training on officer-level outcomes, no instruments to measure these five concepts have been validated in the law enforcement population. The purpose of the current study was to test the reliability and validity of newly designed or adapted measures for the examination of the constructs of self-efficacy, referral decisions, de-escalation skills, attitudes toward psychiatric treatment, and social distance.

2. Methods

2.1. Setting and Samples

Data for testing the initial reliability and validity of the four self-report measures were collected from non-CIT (n=109) and CIT (n=68) police officers attending week-long trainings in six areas throughout the state. The non-CIT group was comprised of officers from varied municipalities who were attending training classes being conducted at the state’s public safety training center on topics such as sexual assault, domestic violence, and supervision. The CIT group was comprised of officers recruited from several CIT training classes being conducted at their respective municipalities.

2.2. Procedures

The protocol was approved Emory University’s Institutional Review Board, and all participants gave written informed consent. The research staff enrolled officers at the beginning of their training week (Monday morning), and participating officers completed the survey before their training program started. Completion of the self-administered, anonymous survey, in which the four measures were embedded, required approximately one hour. A section of the survey was completed by respondents after reading a brief vignette scenario describing an individual with a serious mental illness, excerpts from which are shown in Table 1. The vignette was first developed by the research team and then reviewed by key consultant officers to ensure face validity in terms of real-world situations encountered in the field by officers. Three of the instruments used in this study were rated by respondents after reading the aforementioned vignette. One of the instrument used, Opinions about Psychiatric Treatments (OPT), was not linked to the vignette. The survey was then re-administered to the officers at the end of the training (Friday afternoon), so that matched data were available from both Monday morning and Friday afternoon assessments.

Table 1.

Excerpt from the Vignette Describing an Individual with a Serious Mental Illness

David (while throwing contents of the trashcan around the area): “I know it’s in here somewhere. That thing has been tracking me for 6 months. They think they can control my stomach, but I’m not going to let it continue. I’ll kill them! The CIA is going down! I’ve finally found them!

David then turns his head and listens, as if hearing someone talking from a distance. Then, he talks back to the voices: “I hear that fat chat smat plat. I hear your chatting. You’re probably getting nervous now that I’ve found your shop! Well shut up and stop your chatting!”

Officer pulls onto the scene, with siren and lights on.

Officer gets out of the car.

Officer: “What’s going on here?”

David: “I found this CIA headquarters hidden away here. I’m glad you’re here to arrest them!”

2.3. Materials

2.3.1. Self-Efficacy Scale (SES)

The Self-Efficacy Scale (SES) was modified from a prior study (Bahora et al., 2008). The items were adapted to follow a written vignette describing a person with a mental illness. Responses to the 16 self-report items are given on a 4-point Likert scale ranging from 1=“not at all confident” to 4=“very confident.” Total scores range from 16 to 64, with a higher score indicating a higher level of officers’ confidence in interacting with subjects with a mental illness. Sample questions include: “How confident would you feel talking to someone like David about his mental health treatment?” and “How confident would you feel getting someone like David to talk to you rather than acting out?”

2.3.2. Behavioral Outcomes Scale (BOS)

This 16-item instrument was designed to measure two core behaviors: de-escalation skills and referral decisions. Eight items were written to correspond to each behavior. Officers’ opinions on the effectiveness of specific actions in the situation are rated on a 4-point Likert scale ranging from 0=“very negative” to 3=“very positive.” Total subscale scores range from 0 to 24. Like two of the other three rating scales, the BOS was completed in response to a vignette depicting a scenario that could realistically be encountered by patrol officers. Sample items include “Keeping some space between you and David while you talk to him.” and “Arresting David for disorderly conduct.”

2.3.3. Opinions about Psychiatric Treatments (OPT)

This 20-item measure was developed by the investigative team to complement two additional scales: Opinions about Mental Illnesses (OMI; (Cohen and Struening, 1962; Struening and Cohen, 1963) and Community Attitudes toward the Mentally Ill (CAMI; Taylor et al., 1979; Taylor and Dear, 1981), though these two scales were not used in this reliability/validity study. The new scale was designed to thoroughly assess attitudes about psychiatric treatments more broadly, in addition to hospitals and community mental health treatment facilities. Thus, the OPT assesses attitudes about psychopharmacotherapy, psychotherapy, and psychosocial interventions such as day treatment programs, residential facilities, and case management. Responses are given on a 6-point Likert scale ranging from 1=“strongly disagree” to 6=“strongly agree.” The possible range of scores is 20–120. Sample items include “More tax money should go to support residential facilities for people with serious mental illnesses.” and “Day treatment programs may help people with serious mental illnesses recover.”

2.3.4. Adapted Social Distance Scale (ASDS)

Adapted versions of the SDS (Bogardus, 1925) have been used in the past by the investigative team in several studies (Bahora et al., 2008; Compton et al., 2006). Again, responses were given in relation to the vignette describing an individual with a mental illness. Sample items include “Six months from now, when David is not in crisis, how willing would you be to live next door to him?” and “Six months from now, when David is not in crisis, how willing would you be to have him marry into your family?” Participants were asked to rate their willingness on a 4-point Likert scale ranging from 1=“very willing” to 4=“very unwilling.” Total scores range from 9 to 36. Lower scores indicate lower stigma. In light of past experience with this population, the middle, neutral response option was removed because some officers consistently over-endorsed it, likely to avoid giving socially undesirable responses. Furthermore, in this adaptation of commonly used social distance measures, respondents were asked about “Six months from now, when David is not in crisis…” because the investigative team sought to measure stigma toward an individual with a known history of a mental illness, rather than stigma that might be primarily driven by the severity of the acute crisis event.

2.4. Data Analyses

All variables were initially examined for distribution and variability characteristics. Basic sociodemographic variables of the two groups of officers (non-CIT and CIT) were summarized. Internal consistency reliability analyses for the four instruments (SES, BOS, OPT, and ASDS) were conducted by computing the Cronbach’s α coefficient, and separate coefficients were calculated for subscales where applicable (i.e., BOS). Specifically, Cronbach’s α coefficients for all scales were first calculated based on scores of the two officer groups (CIT and non-CIT), both on the Monday and Friday test administrations; then, a summary coefficient α was calculated for the aggregate sample (CIT and non-CIT combined) for the Monday test administration. Test-retest reliability of all measures was examined by calculating correlations between the Monday and Friday test scores for non-CIT officers, using Pearson product-moment correlation coefficients. In order to assess construct validity (as evaluated through the instruments’ sensitivity to change), pre-test and post-test scores of CIT officers were compared using paired samples t-tests to examine expected differences related to CIT training.

Finally, construct validity was also assessed using Pearson product-moment correlations between scores from the four measures and the 33-item Multiple-Choice Knowledge of Mental Illness Test (MC-KOMIT; Compton et al., unpublished results), based on a priori hypotheses, in the combined Monday sample. The MC-KOMIT is a recently developed knowledge test that was designed as part of the same study, though reported on separately in terms of its internal consistency, test-retest reliability, and construct validity (Compton et al., unpublished results). In the present analysis, it was hypothesized that BOS de-escalation skills, BOS referral decisions, and OPT attitudes would be moderately but significantly positively correlated with knowledge scores derived from the MC-KOMIT. All analyses were conducted using the SPSS 16.0 statistical software package, with the two-tailed criterion for significance set at p<0.05.

3. Results

3.1. Sociodemographic and Work Characteristics of the Study Samples

Among all participants, the mean age was 38.3±8.4 (range=2166), and the mean years of education was 13.8±1.7 (range=1018). A majority of the participants were male (162, 81.4%), reported non-Hispanic ethnicity (187, 94%), and were White (135, 68.5%). The mean number of years of police service was 10.9±7.2 (range=0.530). The separate sociodemographic characteristics of the two groups of officers (non-CIT and CIT) are shown in Table 2.

Table 2.

Sociodemographic Characteristics of the Two Subgroups of Police Officers

Characteristics	Non-CIT Officers (n=109)		CIT Officers (n=68)
	mean	SD	mean	SD
Age, years	38.5	7.1	38.6	9.1
Educational attainment, years	13.7	1.7	14.1	1.7
Years having served as a police officer	12.2	6.8	9.2	6.9
	n	%	n	%
Gender, male	91	84	53	78
Ethnicity, non-Hispanic	106	97	63	96
Race
White/Caucasian	83	76	43	63
Black/African American	20	18	24	35
Other	6	6	1	2
Relationship status
Single and never married	6	6	15	22
Married or living with a partner	78	72	42	62
Separated or divorced	25	23	11	16
Yearly household income
<$40,000	21	19	14	21
$40,000–$60,000	19	17	18	27
>$60,000	69	63	36	53

3.2. Descriptive Characteristics and Initial Reliability and Validity of the Four Measures

3.2.1. Self Efficacy Scale (SES)

The mean scores for the non-CIT group were 48.33±8.06 and 49.63±6.89 on Monday and Friday respectively (range=2264), as shown in Table 3. The scale’s items exhibited high internal consistency reliability in both groups, with Cronbach’s α values of 0.94 and 0.92 for the non-CIT group; and 0.89 and 0.95 for the CIT group, respectively. The Cronbach’s α value for the entire sample on Monday was 0.92. Non-CIT officers’ Monday and Friday scores were significantly correlated at r=0.86, indicating good five-day test-retest reliability. The SES displayed good sensitivity-to-change validity for the CIT group, with mean scores significantly increasing from 48.36±6.5 on Monday to 57.59±6.61 on Friday (t=9.65, df=65, p<0.001). As another approach to construct validity, the SES scale score was found to be significantly, though modestly, correlated with the BOS referral decisions subscale score (r=.35, p<0.001).

Table 3.

Mean Scores, Cronbach’s α Values, Test-Retest r, and Paired t, for OPT, SES, ASDS, BOS (De-escalation), and BOS (Referral Decisions) Scales

	1 n=105	2 n=73	3 n=59	4 n=71	test-retest r (1, 3) paired samples t (2, 4) summary α (1, 2)
Self-Efficacy Scale 16 items, possible range: 16–64	22–64 48.33±8.06 α = 0.94	28–64 48.36±6.5 α = 0.89	35–64 49.63±6.89 α = 0.92	42–64 57.59±6.61 α = 0.95	r = .86** t = −9.65, df=65, p < .001 α = 0.92
Behavioral Outcomes Scale (de-escalation subscale) 8 items, possible range: 0–24	8–22 16.67±2.78 α = 0.42	10–24 17.01±2.83 α = 0.48	10–24 17.09±2.96 α = 0.60	10–23 18.21±2.99 α = 0.55	r = .75** t = −3.29, df=62, p=.002 α = 0.45
Behavioral Outcomes Scale (referral decisions subscale) 8 items, possible range: 0–24	9–24 19.52±3.01 α = 0.71	8–24 19.5±3.21 α = 0.79	13–24 19.61±3.17 α = 0.78	10–24 20.85±3.2 α = 0.82	r = .63** t = −3.89, df=64, p < .001 α = 0.74
Opinions about Psychiatric Treatments 20 items, possible range: 20–120	64–101 81.63±7.17 α = 0.64	59–100 82.58±7.22 α = 0.59	68–106 80.48±7.80 α = 0.71	71–111 88.67±8.60 α = 0.74	r = .79** t = −5.69, df=67, p < .001 α = 0.60
Adapted Social Distance Scale 9 items, possible range: 9–36	9–36 25.25±5.48 α = 0.89	16–34 25.42±4.82 α = 0.86	13–35 25.04±5.34 α = 0.90	9–33 20.25±5.11 α = 0.87	r = .55** t = 7.98, df=67, p < .001 α = 0.88

1 = Monday administration to the non-CIT sample; 2 = Monday administration to the pre-CIT sample; 3 = Friday administration to the non-CIT sample; 4 = Friday administration to the post-CIT sample

3.2.2. De-Escalation Skills Subscale of the Behavioral Outcomes Scale

As shown in Table 3, mean scores for the BOS de-escalation subscale were 16.67±2.78 and 17.09±2.96 for the non-CIT group on Monday and Friday, respectively (range=824). This subscale exhibited relatively low internal consistency in both groups, with Cronbach’s α coefficients of 0.42 and 0.60 for the non-CIT group, and 0.48 and 0.55 for the CIT group on Monday and Friday, respectively. The Cronbach’s α value for the entire sample on Monday was again relatively low (0.45). The test-retest reliability for the non-CIT group for this scale was acceptable (r=0.75). For the CIT group, the mean scores significantly increased from 17.01±2.83 to 18.21±2.29 post-training (t=3.29, df=62, p=0.002), indicating good validity of the measure. As hypothesized a priori, the BOS de-escalation subscale score was correlated with knowledge about mental illnesses, as indicated by the MC-KOMIT score (r=.44, p<0.001), meaning that greater de-escalation skills were associated with higher scores on the knowledge test.

3.2.3. Referral Decisions Subscale of the Behavioral Outcomes Scale

Mean scores for the BOS referral decisions subscale for the non-CIT group were 19.52±3.01 and 19.61±3.17 for both days (range=924), as shown in Table 3. Cronbach’s α values for the CIT group on both days were 0.79 and 0.82 respectively. The subscale also showed acceptable internal consistency reliability in the non-CIT group, with Cronbach’s values of 0.71 and 0.78 on Monday and Friday respectively. The Cronbach’s α value for the entire sample on Monday was 0.74. Scores on Monday and Friday in the non-CIT group were moderately correlated at r=0.63, indicating adequate, though not superior, test-retest reliability. The CIT group’s scores significantly increased from 19.5±3.21 to 20.85±3.2 (t=3.89, df=64, p<0.001), indicating good sensitivity-to-change validity of the measure for this group. Again, this BOS subscale score was significantly associated with the MC-KOMIT score (r=0.32, p<0.001), indicating that better referral decisions were associated with higher scores on the knowledge test.

3.2.4. Opinions about Psychiatric Treatments (OPT)

As shown in Table 3, the mean OPT scores for the non-CIT group were 81.63±7.17 and 80.48±7.80 (range=64106) for the Monday and Friday test administrations, respectively. Cronbach’s α values for the CIT group on both days were moderate at 0.59, and 0.74, respectively. These values for the non-CIT group were 0.64 and 0.71, respectively. Cronbach’s α for the entire sample was 0.60 at the Monday assessment. Non-CIT officers’ Monday and Friday scores were significantly correlated (r=0.79), indicating good test-retest reliability of the measure. For the CIT group, scores significantly increased from 82.58±7.22 to 88.67±8.60 (t=5.69, df=67, p<0.001), indicating good validity of the measure. As predicted, the OPT score was positively associated with the MC-KOMIT knowledge score (r=0.36, p<0.001) and negatively correlated the ASDS social distance stigma score (r=−0.28, p<0.001), in the expected directions.

3.2.5. Adapted Social Distance Scale (ASDS)

The mean scores for the ASDS in the non-CIT group were 25.25±5.48 and 25.04±5.34 for both days, respectively (range=935), as shown in Table 3. The scale displayed good internal consistency for the non-CIT group, with Cronbach’s α values of 0.89 and 0.90, respectively. Cronbach’s α values for the CIT group were 0.86 and 0.87, respectively, for Monday and Friday administrations. The Cronbach’s α value for the entire sample on Monday was 0.88. Non-CIT officers’ scores were moderately correlated at r=0.55, indicating moderately acceptable test-retest reliability. Mean scores for the CIT group significantly decreased from 25.42±4.82 on Monday to 20.25±5.11 on Friday, indicating sensitivity to change after the week of CIT training (t=7.98, df=67, p<0.001). As noted previously, the ASDS scale score was significantly correlated with the OPT score (r=−0.28, p<0.001), and it was also inversely correlated with the SES scale score (r=−0.29, p<0.001).

4. Discussion

Due to ongoing recognition of the criminalization of individuals with mental illnesses, researchers are increasingly exploring such individuals’ interactions with law enforcement officers, especially related to important domains such as officers’ self-efficacy, referral decisions, de-escalation skills, attitudes toward psychiatric treatment, and social distance stigma. The current study developed and tested four instruments measuring these constructs in a sample of police officers. Analyses revealed that the SES, BOS, OPT, and ASDS demonstrated acceptable internal consistency reliability, test-retest reliability, and construct validity, though the BOS de-escalation subscale was found to have low internal consistency reliability. The latter finding suggests that the eight items of that subscale measure diverse constructs, not a unitary de-escalation skills construct. Future research should assess other ways to consider de-escalation subscale scores (i.e., a factor analysis). For example, the de-escalation subscale may be measuring the two constructs of verbal de-escalation skills and physical de-escalation skills.

Due to the dearth of psychometrically sound instruments for research on interactions between police officers and individuals with mental illnesses, valid and reliable measures are needed (Bahora et al., 2008; Bogardus, 1925). The development of the OPT and BOS, and the adaptation of the SES and SDS, provide four new brief, self-administered instruments toward this goal. In the future, these measures may prove useful for researchers examining levels of stigma and self-efficacy in law enforcement communities. Due to the nature of their work, police officers must be equipped to interact with individuals with mental illnesses in their daily work; as such, measures tapping pertinent constructs are vital (Bahora et al., 2008). In terms of practical applications of the results of this study, in addition to the academic research community, law enforcement agencies may find these instruments useful. Police departments implementing CIT —and related programs aimed at improving interactions with and outcomes of individuals with mental illnesses—may benefit from brief, easy-to-conduct evaluations to ensure that they are achieving goals set at initial implementation. All too often, programs with such goals in mind are implemented but not evaluated to demonstrate effectiveness, which may be vital to ensure sustained support and program funding.

Several methodological limitations should be considered when interpreting the results of the current study. First, these measures were designed/adapted for a specific population and therefore may not be generalizable to other samples. Second, due to the low internal consistency reliability of the BOS de-escalation subscale, future analyses should consider alternative ways of handling scores, aside from a summation of scores across the individual items. In addition, criterion validity could not be assessed due to a lack of “gold standard” measures used with police officers with respect to individuals with mental illnesses.

The current study demonstrates that measures can be utilized to reliably and validly examine certain officer-level attitudes and intended behaviors related to interactions with individuals with mental illnesses. Further research should examine these four scales in other samples of officers and test their utility in departmental evaluation contexts. Future scale development should focus on additional attitudinal and behavioral domains, as the present study focused on instrument development related to only five particular constructs. Assessment of such variables pertinent to interactions between police officers and individuals with mental illnesses would be of great importance, due to the wide-reaching implications of such encounters.