The Comparability of the Visual and Verbal Versions of the Inclusion of Community in Self Scale

Abstract

Aims:

Many factors affect the utility and practicality of measures in longitudinal studies characterized by transient participants such as those caught in the cycle of incarceration. The current study evaluated the psychometric equivalency of a visual and a verbal version of a single-item connectedness measure; the aim was to determine whether the different formats can be used interchangeably depending on feasibility.

Methods:

Participants were 133 jail inmates (49% male; 43% Black; M_age=35 years, SD=10 years) interviewed just prior to release from jail.

Results:

Results provide evidence for the concurrent, convergent, and discriminant validity of the two ICS versions. Attempts to calibrate the verbal measure to the visual measure were moderately successful.

Conclusion:

Taken together, results suggest the two formats are comparable, but not interchangeable; they map on to other variables in similar ways but cannot be used in lieu of one another.

Many factors affect the utility and practicality of measures in longitudinal studies characterized by large and transient participant pools. In some cases, it is not possible to use the same assessment format for all participants across time points. For example, high-risk populations like those involved in the criminal justice system are difficult to repeatedly interview in the same setting given intermittent periods of re-incarceration and frequent relocation. Limited access to technology also influences assessment utilization. This challenge raises the question of whether different formats of an assessment (e.g., visual vs. verbal) can be used interchangeably depending on feasibility. The current study employs data from a longitudinal study of jail inmates to examine the psychometric equivalency of a visual assessment, the Inclusion of Community in Self (ICS) scale (Authors, 2007a), and its verbal adaptation.

The Inclusion of Community in Self Scale

The ICS (Authors, 2007; see supplemental material) was derived from the Inclusion of Others in Self (IOS) scale (Aron, Aron & Smollan, 1992), a widely-used measure of connectedness within close relationships. Close relationship researchers developed the IOS under the auspices of the self-expansion model (Aron, Lewandowski, Mashek, & Aron, 2013), which argues individuals are motivated to increase their efficacy in the world and one way they do so is through relationships in which they come to take on the resources, perspectives, and identities of the other. This “taking on” of the other is referred to as including the other in the self.

The IOS is a widely used single-item pictorial measure of self-other overlap. The IOS uses a series of six pairs of circles, with each overlapping slightly more than the preceding pair (parallel to ICS images, see supplemental material). Respondents select the pair that represents their relationship with a person or group. When asked to describe the meaning of the overlapping circles in the IOS validation study, 86% of participants generated connection-themed descriptions. The IOS is a robust predictor of close relationship functioning. For example, a meta-analysis (Le, Dove, Agnew, Korn, & Mutso, 2010) found the IOS was the leading longitudinal predictor of non-marital relationship dissolution. The IOS has been adapted to assess connectedness to relatives and close friends (Uleman, Rhee, Bardoliwalla, & Semin, 2000), coaches, teams, and games among athletes (Blanchard, Perreault, & Vallerand, 1998), and the natural environment (Schultz, 2000); variations use a similar sequence of overlapping circles, with the main difference being in the connectedness target.

Building on evidence of the IOS’s utility and flexibility, researchers working with incarcerated populations created the ICS (Authors, 2007) to assesses connectedness to one’s family, the community at large (CAL), and the criminal community (CC). Participants indicate which pair of circles best describes their actual relationship to these people or groups and in a parallel question, their desired relationship to these same targets.

A growing body of literature provides evidence for the reliability and validity of the visual ICS in college and incarcerated samples (Authors, 2006, 2007). Construct validity has been demonstrated for the CAL item among college students, with the expected moderate correlations with relevant facets of Obst, Smith, and Zinkiewicz’s (2002) Psychological Sense of Community scale: ties and friendship (r = .39), support (r = .28), belonging (r = .27), and conscious identification (r = .45). This item also positively correlated with helping behavior (e.g., picking up a piece of garbage on the sidewalk and throwing it away) and negatively with antisocial behavior (e.g., deliberately damaging someone’s property). The connectedness to the CAL item also demonstrated discriminant validity based on minimal correlations with agreeableness and impression management. Inadequate variance on the criminal connectedness item precluded validity analysis in the college sample.

Further evidence for validity of the visual ICS comes from Time 1 (shortly upon incarceration) of our longitudinal study of jail inmates (Authors, 2007). Connectedness to the CAL upon incarceration correlated positively with kindness/generosity and negatively with antisocial behavior (Authors, 2007), whereas connectedness to the CC positively correlated with criminogenic cognitions and negatively with character strengths (Authors, 2006). Neither correlated with conceptually irrelevant factors such as mania and somatic complaints (Authors, 2007). Analysis of data from Time 2 (just prior to release) of the same study provides evidence for the predictive validity of the ICS (Authors, 2016). Connectedness to the CC assessed with the visual ICS just prior to release predicted recidivism in the first year post-release; pre-release connectedness to the CAL predicted community adjustment during the first year post-release. Thus, the IOS and its derivatives, including the ICS, have proven robust and flexible assessments of connectedness, a construct centrally and critically involved in theories and applications ranging from close relationships, community engagement, social identity, and criminal rehabilitation.

The visual formats of the ICS and IOS are appealing due to their intuitive nature; ability to overcome factors such as language barriers, illiteracy, and social desirability; and the possibility for easy adaptation to assess alternative targets. Yet visual instruments cannot be used in all assessment contexts, such as when in-person interviews are not possible or when participants have visual impairments. For these reasons we devised a verbal parallel to the ICS where participants rate connectedness on a Likert scale. Although asking participants to rate their connectedness is presumed to assess the same construct as the visual ICS, this has yet to be studied empirically. The current study seeks to determine whether the two versions of the ICS are equivalent to warrant their interchangeable use.

Prior Comparisons of Visual and Verbal Assessments

Scant research has compared visual and verbal assessment forms. One exception is within assessment of pain, where there is some evidence that visual and verbal assessments are equivalent and therefore interchangeable among adults (Bijur et al., 2003; DeLoach et al., 1998); among children, they are comparable (i.e., can be likened to one another; similar), but not equivalent or interchangeable (Bailey, Daoust, Doyon-Trottier, Dauphin-Pierre, & Gravel, 2010). Outside the pain literature, we located one published study comparing single-item visual and verbal assessments of anxiety (Davey, Barratt, Butow, & Deeks, 2007). On the visual version, participants rated their current anxiety on a 10-cm horizontal line anchored by “not at all anxious” and “extremely anxious.” On the verbal version, participants rated their current anxiety on a five point Likert scale with the same anchors. The authors did not statistically compare the two assessments, but compared them each to the State form of the Spielberger State Trait Anxiety Inventory and observed high correlations (visual r = 0.78; verbal r = 0.75).

The scant research available suggests visual and verbal measures are generally comparable, and perhaps even equivalent and thereby interchangeable, in some populations. To our knowledge, no research within the social sciences has compared visual and verbal versions of an assessment, so it is unclear whether this conclusion generalizes more broadly.

Current Study

The goal of the current study is to evaluate the equivalency of the visual and verbal adaptation version of the ICS. The visual ICS was developed for use in a longitudinal study of jail inmates (e.g., Authors, 2006, 2007a, 2016) and a verbal version was developed for use during post-release phone interviews. We evaluate the equivalency of these two ICS versions through examination of construct, concurrent, and discriminant validity on a subset of inmates who completed both versions of the measure. We also examine the generalizability of the relations between the connectedness items across demographic factors (i.e., sex, race, age, marital status, educational attainment). Constructs examined for concurrent and discriminant validity were chosen to parallel those used in the initial visual ICS validation at Time 1 (Authors, 2006, 2007); we anticipated similar relations to those found in these studies (e.g., positive correlation between actual connectedness to the CC and criminogenic thinking; positive correlation between actual connectedness to the CAL and character strengths; no relation between connectedness items and mania), as described above. As a secondary aim, we attempt to calibrate the scores of the measures to determine whether a verbal score can be converted into a visual score to use the assessments interchangeably.

Method

Participants and Procedures

Participants were 133 pre- and post-trial inmates held in a metropolitan county jail (49% male), drawn from a larger on-going longitudinal investigation (Authors, 2007b). Participants were 35 years old on average (SD = 10.47 years, range = 18–70 years), racially diverse (43% Black, 39% White, 6% Hispanic, 3% Asian), and had completed an average of 12 years of education (SD = 2.05 years, range = 5–18 years). Eligible inmates were: (a) sentenced to a term of 4 months or more or arrested and held on at least one felony charge other than a probation violation, without bond or on total bond greater than or equal to $7,000; (b) initially assigned to the jail’s medium or maximum security “general population” (e.g., not in segregated housing); (c) had sufficient language proficiency to complete study protocols in English or Spanish (5%); and (d) were at least 18 years old. To avoid the potential confound of language spoken, we did not include Spanish-only speaking participants in the current analysis.

Shortly after assignment to general population, eligible inmates were presented with a description of the study and informed of the voluntary and confidential nature of participation, protected by a Certificate of Confidentiality from the Department of Health and Human Services. The affiliated university Institutional Review Board approved all procedures. Baseline assessment involved 4 to 6 face-to-face and computer-based sessions lasting between 45 and 90 minutes each (Time 1). The touch screen computer required minimal familiarity with computers and presented questionnaires visually and aloud. Research assistants were present to address questions or concerns. Participants received a $15 to $18 honorarium.

Inmates incarcerated in the jail for 6 weeks or more were eligible to complete an additional assessment prior to release into the community, transfer to another correctional facility and/or prior to release from the transfer facility into the community (Time 2). At Time 2, 67% of participants were released into the community and the rest were transferred to other correctional facilities (e.g., federal or state prison for those with sentences greater than 1 year; other jails for those with charges pending in other jurisdictions) prior to release into the community. These face-to-face assessments included many of the intake measures and typically lasted 1 to 2 hours. Participants received a $25 honorarium.

Only a subset of the Time 2 sample (n = 135) was eligible for inclusion in the current analyses because the verbal version of the ICS was introduced approximately one year into the Time 2 follow-up process. Of these participants, one was excluded because they spoke only Spanish and one was excluded based on elevated validity scales (inconsistency and infrequency) on the Personality Assessment Inventory (PAI; Morey, 1991) during the Time 2 interview. Individuals (N = 133) included in the current sample were significantly older, t(493) = −3.47, p = .001, and more likely to be female, χ² = 39.10, p < .001, than those not included in the current sample. No significant differences emerged on race, years of education completed, IQ, or any of the Time 1 visual ICS scores.

Measures

This report draws on the following measures assessed shortly upon incarceration (Time 1) and just prior to transfer or release from jail (Time 2). When applicable we provide reliability estimates from the larger sample of inmates from which these data were drawn, as these are a more robust estimate of measurement consistency.

Time 1. Demographics and Estimated IQ.

Participants self-reported sex, race, age, marital status and educational attainment. The Wonderlic (1999) Personnel Test & Scholastic Level Exam, a widely used brief measure of intelligence, was used to estimate IQ.

Time 2. Visual and Verbal Versions of ICS and Construct Validity Measures.

Visual ICS.

Each dimension of connectedness is assessed using a single-item consisting of six equal sized pairs of overlapping circles. Participants were instructed to “circle the picture that best describes your relationship” (for actual connectedness) or “your desired relationship” (for desired connectedness) to a specific target. Participants rated connectedness to their current family, the CAL (“all the people in your town, city, or county; people in general; people who live on the outside and who do not commit crimes”), and the CC (“people who commit crimes whether they are in jail, prison, or living on the outside”).

Verbal ICS.

The verbal ICS is a 6-item questionnaire assessing connectedness to the same targets as the visual ICS using a 6-point Likert-scale (1 = not at all connected to 6 = as connected as possible). The same definitions for each of the targets were provided as in the visual version. This assessment has yet to be empirically examined.

Moral emotions, criminogenic cognitions, and character strengths.

Shame- and guilt-proneness were assessed using the Test of Self-Conscious Affect for Socially Deviant Populations (TOSCA-SD; Hanson & Tangney, 1996). Using a 5-point scale (1 = not at all likely to 5 = very likely), participants rated how likely they are to react in each way in response to 13 brief scenarios. Each variable was residualized to take into account the other (e.g., “shame-free guilt”). The TOSCA-SD has demonstrated reliability and validity in studies of prison inmates (Hanson, & Tangney, 1996), as well as at Time 1 of the current study (alphas = .71 and .80 for shame and guilt respectively (Authors, 2011).

The 40-item Criminogenic Cognitions Scale (CCS; Authors, 2012) taps five dimensions: notions of entitlement, short-term orientation, negative attitudes towards authority, failure to accept responsibility, and insensitivity to the impact of crime. Responses are rated on a 4-point scale (1 = strongly disagree to 4 = strongly agree). Authors (2012) present evidence for the reliability and validity at Time 1 of the current study (total score alpha = .81; subscale alphas ranged from .51 to .75).

Character strengths were assessed using the Values in Action Inventory of Strengths (VIA-IS; Peterson & Seligman, 2001). Eleven select subscales assessed originality, hope/optimism, capacity to love, industry/perseverance, judgement, integrity/honesty, kindness/generosity, gratitude, spirituality, modesty/humility, and forgiveness. Participants answered four to eight items for each subscale on a 5-point scale (1 = not at all like me to 5 = very much like me). In the full baseline sample, subscale alphas ranged from .74-.86

Self-esteem and Psychological distress.

Self-esteem was assessed using the Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1965). Participants rated 10 statements on a 5-point scale (1 = always false to 5 = always true). The RSE demonstrates Guttman scale coefficient of reproducibility of .92, showing excellent internal consistency (Rosenberg, 1979). Test-retest reliability correlations of .85 and .88 over a period of 2 weeks indicate excellent stability. The RSE correlates significantly with other measures of self-esteem, including the Coopersmith Self-Esteem Inventory (Rosenberg, 1979). Authors (2011) present evidence for the reliability and validity of the RSE at Time 1 of the current study (alpha = .87).

The PAI (Morey, 1991) is a 344-item self-report inventory used to evaluate psychopathology in adults. Participants rate a series of statements on a 4-point scale (1 = false, not at all true to 4 = very true). The depression, anxiety, somatic complaints, mania, drug problems, alcohol problems, and antisocial behavior subscales were used to mirror prior ICS validation studies (Authors, 2006, 2007). The PAI uses T-scores (standardized scores with M = 50 and SD = 10) based on a normative sample and the scales were reliable and valid in the standardization (Morey, 1991) and correctional (Edens & Ruiz, 2005) samples. Authors (2009) present evidence for the reliability and validity of the PAI assessed at Time 1 of the current study (subscale alphas ranged from .83 to .91).

Plan of Analysis

Analyses were conducted in SPSS. Descriptive statistics are shown in Table 1. Concurrent validity was examined by correlating each verbal ICS item with its’ visual ICS counterpart. Linear regression moderation analyses were used to examine generalizability of these correlations across demographic factors; continuous variables were mean centered prior to computing interaction terms. Convergent and discriminant validity of the verbal ICS was examined through correlations with conceptually relevant (i.e., shame- and guilt-proneness; criminogenic cognitions; character strengths; mental health: antisocial, alcohol, drug, depression; self-esteem) and irrelevant (i.e., mental health: anxiety, mania, somatization) factors. The test of the difference between dependent correlations (Lee & Preacher, 2013) was used to determine whether the visual and verbal relations with these constructs significantly differed from one another. Results of these analyses are displayed in Tables 2 and 3.

Table 1.

Descriptive statistics for variables of interest

	N	M	SD	Range	Skewedness	Kurtosis
IQ	123	94.44	14.61	69.00–128.00	0.26	−0.75
Shame-proneness	116	0.05	0.67	−1.21–2.02	0.43	−0.20
Guilt-proneness	116	0.17	0.49	−1.53–1.06	−1.09	1.19
Criminogenic Cognitions
Failure to Accept Responsibility	116	2.01	0.50	1.00–3.20	−0.03	−0.27
Notions of Entitlement	116	2.28	0.49	1.00–4.00	0.09	0.57
Negative Attitudes Toward Authority	116	2.34	0.55	1.00–3.80	0.31	0.62
Short-Term Orientation	116	1.91	0.49	1.00–3.00	−0.08	−0.68
Insensitivity to the Impact of Crime	115	1.79	0.51	1.00–3.60	0.50	0.25
Total	116	2.07	0.37	1.12–3.04	−0.23	−0.08
Character Strengths
Originality	114	3.84	0.67	2.29–5.00	−0.16	−0.71
Hope	115	4.01	0.58	2.13–5.00	−0.44	−0.03
Love	115	4.22	0.59	2.43–5.00	−0.73	−0.01
Industry	115	4.01	0.66	2.00–5.00	−0.62	−0.06
Judgment	115	3.74	0.66	1.75–5.00	−0.43	0.52
Honesty	115	4.21	0.63	2.50–5.00	−0.60	−0.34
Kindness	115	4.10	0.62	2.20–5.00	−0.37	−0.47
Gratitude	115	4.21	0.75	1.67–5.00	−1.20	1.35
Spirituality	115	3.99	0.84	1.60–5.00	−0.89	0.12
Modesty	115	3.48	0.62	2.00–4.88	−0.06	−0.21
Forgiveness	115	3.91	0.64	2.20–5.00	−0.11	−0.64
Mental Health
Antisocial	115	63.97	12.03	45.00–100.00	0.82	0.21
Alcohol	115	60.01	17.19	41.00–104.00	0.88	−0.27
Drug	115	72.21	19.02	42.00–112.00	0.28	−1.00
Somatization	115	50.97	10.59	39.00–91.00	1.63	2.86
Anxiety	115	51.83	9.57	35.00–86.00	0.72	0.65
Depression	115	53.24	11.50	36.00–84.00	0.78	0.17
Mania	115	57.06	12.32	34.00–98.00	0.79	0.72
Self-Esteem	115	4.02	0.66	2.30–5.00	−0.47	−0.24

Table 2.

Relationships between actual connectedness and correlates

	Community at Large			Criminal Community			Family
	Visual	Verbal	Difference	Visual	Verbal	Difference	Visual	Verbal	Difference
Shame-proneness	.05	.07	−0.17	.01	−.02	0.37	−.03	.01	−0.40
Guilt-proneness	.05	.16	−1.01	−.14	−.14	0.06	.00	.06	−0.60
Criminogenic Cognitions
Failure to Accept Responsibility	.03	−.05	0.75	.13	.18	−0.56	−.196	−.175	−0.22
Notions of Entitlement	.19	−.02	1.98*	−.04	.03	−0.82	.036	.024	0.12
Negative Attitudes Toward Authority	.07	−.20*	2.59**	−.06	−.09	0.26	.080	.060	0.20
Short-Term Orientation	.05	−.11	1.50	.00	.05	−0.67	−.010	−.118	1.09
Insensitivity to the Impact of Crime	.10	−.03	1.20	.04	.10	−0.63	−.013	−.025	0.12
Total	.12	−.12	2.25	.02	.07	−0.64	−.025	−.059	0.34
Character Strengths
Originality	.22*	.22*	0.02	.14	.11	0.34	.27**	.26*	0.10
Hope	.19*	.24*	−0.45	.00	.01	−0.06	.16	.26*	−1.05
Love	.17	.20*	−0.32	.02	.07	−0.56	.25*	.29**	−0.40
Industry	.19	.20*	−0.13	−.03	.01	−0.45	.17	.29**	−1.28
Judgment	.19	.20*	−0.15	−.14	−.14	0.05	.18	.27**	−0.99
Honesty	.20*	.19	0.07	−.04	−.07	1.38	.28**	.24*	0.42
Kindness	.24*	.20*	0.39	−.09	−.08	−0.14	.21*	.25*	−0.41
Gratitude	.24*	.22*	0.24	−.04	−.11	0.86	.22*	.21*	0.20
Spirituality	.14	.17	−0.31	.01	.01	0.00	.26*	.15	1.14
Modesty	.22*	.25*	−0.25	.08	.02	0.81	.18	.26*	−0.84
Forgiveness	.17	.18	−0.04	−.04	−.11	0.91	.23*	.23*	0.63
Mental Health
Antisocial	−.08	−.15	0.74	.08	.26**	−2.19*	−.28**	−.16	−1.33
Alcohol	.00	−.15	1.42	−.04	.10	−1.69	−.24*	−.14	−1.05
Drug	−.20*	−.09	−0.99	.16	.20*	−0.48	−.11	−.03	−0.82
Somatization	−.06	−.04	−0.14	.00	−.14	1.63	.06	−.04	1.01
Anxiety	−.03	−.15	1.12	−.01	.11	−1.39	−.17	−.27**	1.05
Depression	−.09	−.17	0.73	−.02	.01	−0.39	−.17	−.30**	1.28
Mania	.12	.03	0.87	.01	.22*	−2.61**	−.07	.03	−0.97
Self-Esteem	.12	.20*	−0.78	.00	.04	−0.49	.25*	.21*	0.43

Note.

^**p < .01

^*p < .05

Values under the visual and verbal columns are rs and under the difference score column are z-scores. Bold values represent statistically significant difference scores.

Table 3.

Relationships between desired connectedness and correlates

	Community at Large			Criminal Community			Family
	Visual	Verbal	Difference	Visual	Verbal	Difference	Visual	Verbal	Difference
Shame-proneness	.06	.15	−0.93	.07	−.01	0.98	.07	.03	0.31
Guilt-proneness	.11	−.01	1.29	−.06	−.13	0.86	−.02	.17	−1.38
Criminogenic Cognitions
Failure to Accept Responsibility	.08	.04	0.41	.22*	.24*	−0.20	−.06	−.20	1.08
Notions of Entitlement	−.06	−.12	0.69	.14	.29**	−.193	−.01	−.11	0.68
Negative Attitudes Toward Authority	−.32**	−.36**	0.53	.13	.08	0.65	.04	−.07	0.77
Short-Term Orientation	−.11	.03	−1.50	.22*	.19	0.41	−.04	−.22*	1.36
Insensitivity to the Impact of Crime	−.06	−.05	−0.09	.22*	.27**	−0.64	.04	−.15	1.40
Total	−.14	−.14	0.03	.26*	.29**	−0.43	−.01	−.20	1.43
Character Strengths
Originality	.13	.06	0.85	.11	.25*	−1.70	.11	.21*	−0.77
Hope	.08	.02	0.75	.13	.17	−0.54	−.02	.12	−1.07
Love	.17	.07	1.21	.03	.09	−0.66	.09	.06	0.22
Industry	.62	−.01	0.81	.02	.08	−0.67	−.06	.03	−0.61
Judgment	.01	−.09	1.11	.07	.15	−1.03	−.05	.16	−1.51
Honesty	.08	−.06	1.59	.06	.04	0.25	−.02	.06	−0.57
Kindness	.15	.10	0.63	.02	.03	−0.05	.05	.13	−0.60
Gratitude	.11	.04	0.74	.09	−.06	1.78	.08	.08	−0.05
Spirituality	.09	.11	−0.17	.16	.07	1.13	.16	−.03	1.38
Modesty	.18	.14	0.45	.15	.09	0.75	−.02	−.04	0.09
Forgiveness	.07	.12	−0.54	.03	−.02	0.60	.17.	.25*	−0.60
Mental Health
Antisocial	.01	−.02	0.23	−.03	.17	−2.50*	−.04	−.18	1.02
Alcohol	−.06	.03	−1.06	−.03	.01	−0.45	−.11	−.12	0.10
Drug	.01	.17	−1.88	.04	.05	−0.12	−.04	−.09	0.37
Somatization	−.15	.01	−1.74	−.01	−.08	0.92	.07	.02	0.35
Anxiety	−.10	−.05	−0.51	.08	.13	−0.67	.08	−.11	1.43
Depression	−.08	−.01	−0.74	−.07	−.04	−0.41	.02	−.06	0.55
Mania	.11	.01	1.02	.21*	.38**	−2.27*	−.02	.12	0.68
Self-Esteem	.09	−.04	1.39	.03	.05	−0.26	−.03	.02	−0.32

Note.

^**p < .01

^*p < .05

Values under the visual and verbal columns are rs and under the difference score column are z-scores. Bold values represent statistically significant difference scores.

We followed the procedures outlined by McKnight (1997) to calibrate (Sechrest, McKnight, & McKnight, 1996) the visual and verbal versions of the ICS. Calibration is necessary to permit transformations of values, substitutions of measures, and standardization (Sechrest et al., 1996). To calibrate the two scores, we conducted separate regressions for each target, regressing the visual score on the verbal score. We used the resultant unstandardized regression coefficients to formulate the conversion algorithm: predicted visual score = intercept + (slope x verbal score). Finally, we computed intraclass correlations (ICCs) between the adjusted verbal score and the original visual score.

Results

Concurrent Validity

All actual connectedness items assessing the same target on the visual and verbal versions were significantly correlated (CAL r = .43, p < .001; CC r = .65, p < .001; family r = .58, p < .001). For desired connectedness, significant correlations were observed for the CAL (r = .61, p < .001) and CC (r = .69, p < .001), but not family (r = .11, p = .28). Effect sizes were moderate to large except for desired family connectedness, which showed no relationship, thus not passing the first hurdle assessing concurrent validity. The other items show some evidence of concurrent validity, though less than expected given they intend to assess the same construct.

We next examined five demographic variables (i.e., sex, race, age, marital status, educational attainment) and estimated IQ as moderators of agreement between the visual and verbal assessments. The relationships between the two versions generalized across sex, marital status, and educational attainment. Age moderated three of the six relations: actual connectedness to the CC (β = −.18, p = .011), desired connectedness to the CC (β = −.29, p < .001), and desired family connectedness (β = −.29, p = .009). In all three analyses, younger individuals tended to be more consistent. Race moderated the relation between the desired connectedness to the CC items, (β = .41, p = .003); compared to White participants, Black participants tended to be more consistent. IQ moderated the relation between the desired connectedness to the CAL items, (β = −.19, p = .025); individuals with lower IQ tended to be more consistent. No other significant moderation effects were present.

Convergent and Discriminant Validity

Results are displayed in Tables 2 (actual connectedness) and 3 (desired connectedness) and show some evidence for convergent and discriminant validity. Of note, actual connectedness to the CAL (both visual and verbal) was related positively to character strengths and self-esteem (verbal), and un-related or negatively (verbal) related to criminogenic cognitions and drug problems (visual). Actual connectedness to the CC was positively related to antisocial features, drug problems, and mania (verbal), though contrary to our hypotheses, unrelated to criminogenic cognitions. Actual connectedness to family was positively related to character strengths and self-esteem (visual and verbal), and negatively to antisocial features (visual) alcohol problems (visual), anxiety (verbal), and depression (verbal). Regarding desired connectedness, CAL was negatively related to negative attitudes toward authority (visual and verbal); CC was positively related to multiple criminogenic cognition subscales (visual and verbal), and positively related to originality (verbal) and mania (visual and verbal); family was negatively related to short-term orientation (verbal) and positively related to originality and forgiveness (verbal).

Comparing Relations of Constructs with the Verbal and Visual Assessments: Results after a Benjamini-Hochberg (B-H) correction.

A B-H correction (Benjamini & Hochberg, 1995) was applied to the difference tests to control for familywise error. This procedure has greater power and stability in power as the number of comparisons increases, compared to the Bonferroni procedure (Williams, Jones, & Tukey, 1999). Separate B-H corrections were conducted for each of the six connectedness variables. When a B-H correction is applied, no tests of the difference between dependent correlations remained significant.

Results prior to correction.

The following differences were present prior to the correction and may be relevant for future research. Significant findings are reviewed.

Moral emotions, criminogenic cognitions, and character strengths.

There was a significant difference for actual connectedness to the CAL in its relation to notions of entitlement (Z = 1.98, p = .05), though neither version was significantly related to notions of entitlement. Similarly, there was a significant difference for actual connectedness to the CAL in relation to negative attitudes toward authority (Z = 2.59, p = .01), where the visual version was unrelated (r = .07, p =.47) and the verbal was negatively related (r = −.20, p = .04).

Self-esteem and psychological distress.

There were significant differences for actual connectedness to the CC in its relation to antisocial features (Z = −2.19, p = .03) and mania (Z = −2. 16, p =.01). The visual version was not related to antisocial features (r = .08, p = .41) or mania (r = .01, p = .91) and the verbal version was significantly positively related to both antisocial features (r =.26, p =.01) and mania (r = .22, p = .02). There was also a significant difference for desired connectedness to the CC in its relation to antisocial features (Z = −2.48, p = .01). Neither version was significantly related to antisocial features, but the effects were in opposite directions for the visual (r = −.03, p = .76) and verbal (r = .17, p = .10) versions.

Calibrating Measures Against Each Other

Verbal and visual responses were calibrated using the steps outlined by McKnight (1997). We did not calibrate desired family connectedness due to the low correlation between the items. Table 4 shows the relevant parameters for the conversion algorithms. ICCs estimated based on consistency of scores between the original visual and the adjusted verbal scores based on the calibration equation ranged from poor (actual connectedness to the CAL) to good (desired connectedness to the criminal community) based on Cicchetti and Sparrow’s (1981) criteria (below 0.40 = poor; 0.40–0.59 = fair; 0.60–0.74 = good; 0.75–1.00 = excellent). Although predominately acceptable by this standard, discrepancy between the scores remains and they are not interchangeable.

Table 4.

Calibration statistics

	B	INT	R2	ICC
Actual Connectedness to the Community at Large	.409	1.096	.183	.309
Desired Connectedness to the Community at Large	.599	1.234	.373	.543
Actual Connectedness to the Criminal Community	.597	1.116	.421	.592
Desired Connectedness to the Criminal Community	.658	0.510	.472	.641
Actual Family Connectedness	.700	1.074	.337	.504

Note. ICCs are estimated based on consistency between the actual visual ICS score and the predicted visual ICS score based on the calibration equation.

Discussion

Although visual assessments are intuitively appealing, they are not always practical. The current study examined the psychometric qualities and equivalency of the visual and verbal versions of the ICS. Results provide evidence for the concurrent, convergent, and discriminant validity of the two ICS versions. Calibration of the verbal to the visual version was not successful, indicating the scores cannot be treated as equivalent.

Except for desired family connectedness, there were moderate to large correlations between the visual and verbal assessment items, indicating individuals tend to respond similarly. These relations generalized across most demographic characteristics, although there were some differences, particularly based on age. It is possible young adults are more familiar with visual presentations of scales compared to older adults who did not grow up in the age of technology, completing questionnaires online using multiple formats. As such, special efforts may be needed to explain the visual ratings scales such as the ICS to older adults, and some caution is warranted in interpreting other age-related effects that may simply reflect differential validity.

Both versions of the ICS were similarly related to theoretically relevant constructs and unrelated to theoretically irrelevant constructs, as outlined by Authors (2006, 2007). For example, actual connectedness to the CAL positively correlated with character strengths whereas actual connectedness to the CC was unrelated. One notable difference from Authors (2006, 2007) is actual connectedness to the CC was not related to criminogenic cognitions in the current report, though desired connectedness to the CC was. This may be due to differences in assessment timing - the end of incarceration and therefore a period of embeddedness in the criminal community vs. the start of incarceration (Authors, 2006, 2007). In general, these results replicate findings from the initial ICS validation study conducted with data from Time 1 of the longitudinal study from which the current data were drawn (Authors, 2006, 2007).

Regarding interchangeability, efforts to calibrate were moderately successful, suggesting the two versions are comparable but not interchangeable. This is consistent with prior research examining pain assessments (Bailey et al., 2010; Bijur et al., 2003; DeLoach et al., 1998).

Limitations and Future Directions

Key limitations of the current study are not counterbalancing the two ICS versions (presented in the same interview approximately 30 minutes apart due to the verbal adaptation being slotted into the standardized administration protocol) and the ICS items not being truly continuous variables. To our knowledge there are no truly continuous verbal versions of assessments of connectedness or closeness as conceptualized by the current self-expansion model, though the computerized visual IOS uses a scale from 0 to 100 (Le, Moss, & Mashek, 2007) and might be more successfully calibrated. Future research is needed to replicate the moderating effect of, and to extend these results to community members and to languages other than English.

Summary and Conclusions

Most psychological measures rely on words, yet some constructs may be more readily conveyed via pictures. Inclusion of other(s) in the self (Aron et al., 1992; Aron, et al., 2013) is one such construct and the visual ICS directly captures the metaphor of self-other overlap. We determined Likert ratings “connectedness” cannot be used interchangeably with the visual ICS. Both versions showed convergent and discriminant validity, however, and there was no evidence favoring one mode of presentation.

If the visual and verbal versions are equally valid in a given population, selecting one version over the other may be based on preference or convenience of the researchers. The visual version may be more intuitive for and preferred by some populations with limited language abilities, like children (Bailey et al., 2010). In addition, social desirability may be less influential with visual assessments, where psychological defenses may be more easily bypassed.

Researchers working with hard to reach samples with wide variance in terms of literacy and access to technology know alignment between study designs and assessment options is not always possible. In these contexts, one might understandably lean toward the path of less resistance by tabling certain theoretical and practical questions, or selecting a more convenient sample with which to explore those questions. The better solution in terms of research ethics and the broader impact of our work is to figure out ways of assessing these key constructs despite the challenges of doing so. Efforts to create interchangeable forms of assessment, such as undertaken here, thus hold import in answering the call to extend the reach of psychological science.