Assessing Social Support, Companionship, and Distress: NIH Toolbox Adult Social Relationship Scales

Abstract

Objective

The quality of our daily social interactions – including perceptions of support, feelings of loneliness, and distress stemming from negative social exchanges – influence physical health and well-being. Despite the importance of social relationships, brief yet precise, unidimensional scales that assess key aspects of social relationship quality are lacking. As part of the NIH Toolbox for the Assessment of Neurological and Behavioral Function, we developed brief self-report scales designed to assess aspects of social support, companionship, and social distress across age cohorts. This report details the development and psychometric testing of the adult NIH Toolbox Social Relationship scales.

Methods

Social relationship concepts were selected, and item sets were developed and revised based on expert feedback and literature review. Items were then tested across a community-dwelling U.S. internet panel sample of adults aged 18 and above (N=692) using traditional (classic) psychometric methods and item response theory (IRT) approaches to identify items for inclusion in 5–8 item unidimensional scales. Finally, concurrent validity of the newly-developed scales was evaluated with respect to their inter-relationships with classic social relationship validation instruments.

Results

Results provide support for the internal reliability and concurrent validity of resulting self-report scales assessing Emotional Support, Instrumental Support, Friendship, Loneliness, Perceived Rejection, and Perceived Hostility.

Conclusion

These brief social relationship scales provide the pragmatic utility and enhanced precision needed to promote future epidemiological and social neuroscience research on the impact of social relationships on physical and emotional health outcomes.

There has been a great deal of research on the impact of social relationships on both emotional and physical health (Berkman & Glass, 2000; Cohen & Gottlieb, 2000; Suls & Wallston, 2003; Uchino, 2004). The quality of our daily social interactions, the support we receive from people around us, and our feelings of loneliness influence our health and well-being from childhood through old age. A growing body of research supports the positive health consequences of social support (Cohen, 2004; Kiecolt-Glaser, Gouin, & Hantsoo, 2010; Seeman, 2000) and the deleterious outcomes associated with feelings of loneliness or distress rooted in negative social interactions (Cacioppo et al., 2002; Hawkley, Masi, Berry, & Cacioppo, 2006).

These data underscore the potential importance of measuring social relationship quality within and across studies assessing a range of health outcomes. Yet, many such opportunities are missed or limited by pragmatic measurement issues. Most extant social relationship scales are lengthy and multidimensional; few have been developed for use within population-based samples; and none provide for assessment across the full self-report age range. Thus, future research on social relationships and health would be facilitated by the development of practical assessment scales for measuring key aspects of social relationship quality across the lifespan.

Social Relationship Assessment within the NIH Toolbox Project

The NIH Toolbox for the Assessment of Neurological and Behavioral Function, funded as part of the NIH Neuroscience Blueprint initiative, was designed to support development of brief assessment tools to measure key domains of motor, cognitive, sensory and emotional health in individuals aged 3–85. Four subdomains of emotional health were identified for NIH Toolbox assessment, based on goals articulated in the NIH Request for Proposals and expert feedback elicited through an NIH Expert Request for Information (RFI) and completed by 147 expert opinion leaders (64% of 232 invited experts) from NIH research program staff and funded investigators. Emotional health concepts selected for inclusion were: negative affect, positive affect, stress and self-efficacy, and social relationships (Salsman et al, under review). This report describes development and testing of the adult NIH Toolbox Social Relationship scales.

Primary Concepts Included in the NIH Toolbox Social Relationship Assessment Battery

Literature searches were conducted to identify self-report scales measuring relevant social relationship constructs. Initial searches were run in February 2007 using HaPI (Health and Psychosocial Instruments), PsycINFO, and Medline databases and targeted search strategies crossing assessment-related terms with content-relevant terms describing positive or negative aspects of social relationships (e.g., arguments, conflict, emotional support, friendship, intimacy, loneliness, rejection, relationship satisfaction, social support). This yielded 658 hits (234 in HaPI; 178 in PsycINFO; 246 in Medline). Article abstracts were reviewed by two co-authors (JC and MK), with additional articles identified based on citation searches of key papers.

Selection of social relationship concepts to be included in the NIH Toolbox was based on conceptual and pragmatic factors. First, social relationship concepts that had been empirically linked with broader health outcomes were identified. Given the goals of the NIH Toolbox project, only concepts with relevance across population-based samples (that may vary by age, race, culture, marital/family status, etc.) were included. To facilitate utility across the general population, attempts to assess specific social activities or relationships were rejected in favor of assessing general perceptions regarding the availability or quality of respondents’ current social relationships. Finally, given the goal of assessing aspects of social relationships that may change over time or in response to clinical intervention, individual difference constructs related to socially-relevant traits or enduring personality features (such as introversion/extraversion, shyness, behavioral inhibition or adult attachment patterns) were excluded from consideration.

Based on the above-described results and selection criteria, three primary concepts were identified for assessment: social support, companionship, and social distress. Selection of components (and representative items) to be included within each of these social relationship concepts was guided by the literature and by item-level review, and was engineered to: (1) provide unidimensional component scores that provided reasonable conceptual coverage of each concept; (2) provide multiple items within each component, to allow for subsequent item attrition without losing conceptual coverage; and (3) provide a theoretically-tenable dimensional framework that could be empirically tested in subsequent psychometric analyses. Definitions of the primary SR concepts and components used to guide scale development are provided below.

Social Support

A growing body of evidence supports the notion that supportive social relationships are beneficial for physical and emotional health (Berkman & Glass, 2000; Uchino, 2004; Uchino & Birmingham, 2010). For the NIH Toolbox, social support was defined as the extent to which an individual views his or her social relationships as available to provide aid in times of need (Cohen, 2004). Multiple types of social support have been identified, typically categorized by the form of aid provided. Three types of social support were selected for representation in initial item pools: instrumental, informational and emotional support (Cohen, 2004; House & Kahn, 1985). Instrumental Support refers to the perceived availability of people who can provide functional aid in completing daily tasks (such as making meals or providing transportation) if needed. Informational Support refers to the availability of people who can provide information or advice needed to solve problems that arise. Emotional Support refers to the availability of people to listen to one’s problems with empathy, caring and understanding.

Companionship

The deleterious effects of loneliness on health outcomes have been demonstrated in a variety of samples (Asher, Parkhurst, Hymel, & Williams, 1990; Cacioppo, et al., 2002; Hawkley, et al., 2006). Three components of relevance to the concept of companionship were identified for inclusion in NIH Toolbox, namely friendship, intimacy, and loneliness. Friendship refers to the availability of companions with whom to interact or affiliate. Intimacy refers to the availability of people with whom one feels emotionally close or connected. Finally, Loneliness refers to the perception that one is lonely or socially isolated from others.

Social Distress

An equally important but relatively understudied SR concept relates to distress experienced within or across social relationships (Krause, 1995; Newsom, Rook, Nishishib, Sorkin, & Mahan, 2005; Rauktis, Koeske, & Tereshko, 1995; Schuster, Kessler, & Aseltine, 1990). Research on relationship strain or dissatisfaction (Orth-Gomer et al., 2000) and perceptions of criticism, hostility or emotional over-involvement from relationships [included in the concept of expressed emotion (Brown, Birley, & Wing, 1972; Brown & Rutter, 1966)] supports the deleterious impact of social distress on health outcomes (Hooley & Gotlib, 2000; Kiecolt-Glaser, et al., 2010). For the purpose of the NIH Toolbox, social distress was defined as the extent to which an individual perceives his or her daily social interactions as negative or distressing. Four primary subdomains with relevance to this concept were targeted for assessment. Specifically, respondents were asked to rate how often people in their daily life behave in ways that might be perceived as: (1) Hostile/Critical (e.g., how often people argue with me, yell at me, or criticize me); (2) Insensitive/Neglectful (e.g., how often people don’t listen when I ask for help, or don’t pay attention to me); (3) Rejecting/Ridiculing (e.g., how often people act like they don’t like me, or make fun of me); and (4) Intrusive (e.g., how often people invade my privacy, or make too many demands of me).

Method

Item Selection, Development, and Refinement for NIH Toolbox SR Self-Report Scales

Items representing the above SR concepts were written or adapted from extant scale items that could be cleared for use without payment (for example, Carver, 2006; Tarlov et al., 1998). Items were written or extensively revised to reflect concept definitions, maximize conceptual coverage, optimize relevance across a range of respondents, and simplify language (vocabulary ≤ 6^th grade reading level). All NIH Toolbox items were also reviewed by expert panels to identify assessment issues across special subgroups, including a Geriatric Working Group tasked with reviewing items for use by older respondents (headed by Hugh Hendrie, Christy Purnell, Richard Havlik); a Cultural Working Group tasked with reviewing items for sensitivity, utility and conceptual appropriateness across cultural groups (headed by David Victorson, Jennifer Manly, Helena Correia), and a Spanish Language Working Group (SLWG), tasked with evaluating Spanish translation issues (headed by Helena Correia). Working groups reviewed all self-report items, identified difficulties that might arise when administering items across special populations, and offered alternative wording to be considered in iterative item revisions by the domain team. For social relationship items, all suggested edits deemed not to change item meaning were incorporated. For detail, see Victorson et al (under review).

This work resulted in a total adult SR pool of 97 items, including: 31 social support items (10 Emotional Support, 11 Instrumental Support, and 10 Informational Support items); 32 companionship items (10 Friendship, 10 Intimacy, and 12 Loneliness items); and 34 social distress items (8 Hostility/Criticism, 8 Insensitivity/Neglect, 9 Rejection/Ridicule, and 9 Intrusiveness items). To minimize item complexity, items were written in the present tense; however, respondents were instructed to consider their social relationships as experienced over a past-month time frame. (See Appendix A for scale instructions.) Items were rated on a 5-point frequency scale including: Never (1), Rarely (2), Sometimes (3), Usually (4), and Always (5).

Overview of Initial Psychometric Testing

Subjects and methods

A community-dwelling internet panel sample of 692 adults (see Table 1) was recruited by Toluna, an internet survey company. Quota sampling based on US census data was used to recruit a nationally representative sample with respect to age strata, race and gender (see http://www.toluna-group.com for detail). Respondents completed NIH Toolbox instruments, related validation scales, and an additional demographic survey (for quality assurance procedures, see http://www.greenfield-ciaosurveys.com/html/qualityassurance.htm), and were eligible for prize or incentive-based compensation following survey completion.

Table 1.

Demographics of the adult survey sample (N = 692)

Mean age (SD), in years	43.97 (16.73)
Sex (% male)	43.4
Race (%)
White	82.5
African American	10.7
Asian	2.6
Other	7.0
Hispanic origin (%)	9.4
Marital Status (%)
Never married	27.9
Married	46.0
Living with partner	10.0
Separated/Divorced	12.3
Widowed	3.9
Education (%)
Less than high school	0.5
Some high school	3.7
High school grad	22.5
Some college	44.8
College degree	19.8
Advanced degree	8.5

Note: SD = standard deviation

Psychometric analyses and item selection procedures

Psychometric analyses were conducted separately for social support, companionship, and social distress items, and included evaluation of item and scale properties using classical test theory and item response theory (IRT). The process of item testing and selection occurred over a series of four stages, namely: (1) initial item level evaluation; (2) dimensionality analyses; (3) IRT analyses; and (4) final item selection (including CAT simulations). As part of the initial item level evaluation, items with response option categories endorsed by fewer than 5 people, or those with inversions in mean total scores were identified for exclusion. Inversions in mean total scores (when comparing people across adjacent response options per item) indicate inconsistent use of the rating scale for that item and (like sparse cells) adversely impact IRT analyses. Corrected item-total correlations were also evaluated at this stage to identify aberrant items within each of the three SR domains.

Next, dimensionality analyses were conducted to identify the factor structure of responses to items within each of the three SR concepts. Exploratory factor analyses (EFA) were first run on half the sample (randomly selected) to identify underlying factors for each of the primary SR concepts using Mplus (Muthen & Muthen, 2006). Confirmatory factor analyses (CFA) were subsequently conducted on the second half of the sample to verify the factor structure. Because the data were ordinal in nature, polychoric correlations were used in factor analyses. Unweighted least squares estimation and Quartimin rotation were used in EFAs, and eigenvalues > 1 and scree plots were used to identify the number of factors. To enhance the unidimensional nature of factors, items that displayed weak factor loadings or cross-loadings across factors were excluded. For CFAs, weighted least squares estimation and fit statistics were used to evaluate the dimensionality of the item pool. Fit statistic guidelines used to confirm unidimensionality were: Comparative Fit Index (CFI) > 0.90, Root Mean Square Error of Approximation (RMSEA) < 0.10 representing acceptable fit; and CFI > 0.95, RMSEA < 0.08 representing good fit.

Item response theory (IRT) analyses were subsequently used to guide selection of items to be included in short-form scales. IRT refers to a class of psychometric techniques in which the probability of choosing each item-response category is modeled as a function of a latent trait of interest. IRT models allow investigators to evaluate the performance of individual items, which supports the construction of short scales that provide adequate precision across the range of the latent construct of interest. Data were calibrated using Samejima’s graded response model (Samejima, 1969) to estimate parameters, plotting the information provided by individual items as reported in Multilog (Thissen, Chen, & Bock, 2003). IRT assumptions were tested by assessing fit to a 2-parameter IRT model using the S-X² fit statistic (Orlando & Thissen, 2003) as implemented in IRTfit (Bjorner et al., 2006). Items displaying poor fit (i.e., prob S-X² <.001), poorly discriminating items (i.e., those with IRT slopes < 1), and items providing little information across the scale range (e.g., items with information functions having a peak of less than 0.5) were considered for exclusion at this stage of testing. The latter items typically displayed skewed response distributions.

Final selection of items and short-form scales for inclusion in the SR battery was based on pragmatic factors and CAT simulation data. At this stage, unidimensional scales with few (< 5) items, and items with no “sister item” (i.e, an identical or similar item) included in parallel pediatric scales were considered for exclusion. Finally, post-hoc computerized adaptive testing (CAT) simulations were conducted on remaining items using Firestar (Choi, 2009) with the calibration data. This was implemented by simulating the order in which items would be presented to individuals in the current study sample using CAT-based information parameters. Then, for each item, the rank in which the item was administered was averaged over all simulated administrations. In this way, and in conjunction with content considerations, we were able to select items with the minimum average CAT presentation rank (Min Ave CAT Rank), i.e., those items most likely presented in the early stages of CAT-based adaptive testing.

Initial evaluation of concurrent validity

Additional self-report scales were also included in survey testing, in order to evaluate concurrent validity of the new NIH Toolbox SR scales.

Social Support: Interpersonal Support Evaluation List (ISEL)

(Cohen & Hoberman, 1983; Cohen, Mermelstein, Kamarck, & Hoberman, 1985). The 12-item ISEL assesses perceived availability of various types of social support, including Tangible (material aid), Appraisal (someone to talk to about one’s problems), and Belonging (people one can do things with) support. The ISEL displays good reliability (Cohen et al., 1985; Cohen & Wills, 1985); Cronbach’s alpha in current sample=0.912. The current report utilized the typically-used ISEL total score, as well as 4-item Appraisal, Tangible, and Belonging support subscale scores.

Companionship: Revised UCLA Loneliness Scale

(R-UCLA; Russell D, Peplau LA, & Cutrona CE, 1980). The 20-item revised R-UCLA has been shown to display good reliability (Russell et al., 1980) and concurrent validity (Cacioppo et al., 2000; Hawkley, Burleson, Berntson, & Cacioppo, 2003). Cronbach’s alpha in the current sample=0.949. While primary analyses used the R-UCLA total score, secondary analyses utilized brief composite scores derived from recent factor analytic work by Hawkley et al (2005), who identified three factors across college-age and older cohorts: (1) Isolation (feelings of aloneness or social isolation), (2) Relational Connectedness (feelings of familiarity, closeness or support), and (3) Collective Connectedness (feelings of group identification or cohesion). For exploratory purposes, we developed composite scores using R-UCLA items with highest factor loadings in both young and older adult samples, namely: Isolation (2 items: I feel left out; I feel isolated from others), Relational Connectedness ( 2 items: There are people I can talk to; There are people I can turn to), and Collective Connectedness (3 items: I feel part of a group of friends; I have a lot in common with people around me; I feel in tune with the people around me).

Social Distress: Negative Interaction Scale

(Krause, 1995; Matthews, Owens, Edmundowicz, Lee, & Kuller, 2006; Schuster, Kessler, & Aseltine, 1990). This 5-item scale was adapted from a 4-item scale developed by Krause (Krause, 1995) to assess negative interactions with significant others (Matthews et al., 2006). This scale has been shown to display adequate reliability (Cronbach’s alpha = 0.87) (Krause, 1995) and convergent validity with relationship dissatisfaction (Krause & Thompson, 1997). Cronbach’s alpha in the current sample=0.854.

Results

Item Selection and Scale Development

Item selection results for each of the SR concepts are presented below. Further details regarding specific findings at each step of the item selection process are available as part of Health Psychology’s online supplemental materials.

Social Support Scales

Two factors were identified by in the initial EFA (eigenvalues = 21.74; 1.94) of the split adult sample (N=348). Items from the original Emotional Support and Informational Support item pools loaded on the first factor, while items from the original Instrumental Support pool loaded on the second factor. A 2-factor confirmatory factor analysis (CFA) conducted on the remaining half sample (N=344) indicated acceptable fit (CFI = 0.945, RMSEA = 0.127). For simplicity, factor 1 and factor 2 scales were named Emotional Support and Instrumental Support, respectively. Eight items were eliminated based on IRT analyses, with remaining items subjected to CAT simulation. The eight top-performing items for each factor were selected for final scale inclusion based upon IRT item information curves, CAT-based rankings, item content, and availability of parallel pediatric items. See Appendix A.

Companionship Scales

Three factors were identified by the initial EFA (eigenvalues = 20.75, 1.84, 1.32) conducted on the split sample (N=348). However, CFA indicated that the 3-factor model displayed poor model fit (CFI = 0.846, RMSEA = 0.145). In particular, the first factor displayed unacceptable fit, due to a number of items with weak primary factor loadings and high cross-loadings. Re-evaluation of the 3-factor CFA after dropping these items indicated acceptable model fit (CFI = 0.923, RMSEA = 0.129). Factor 1 included Friendship items and Intimacy items that referenced friendship relationships; this factor was labeled Friendship. Factor 2 included items from the original Loneliness component, and was labeled Loneliness. Factor 3 included only 6 items labeled as Non-Friendship Intimacy, 2 of which were excluded following IRT analyses. Given the limited nature of the Non-Friendship Intimacy item pool, none of which were represented in the pediatric scales, this scale was dropped from the Toolbox battery. Following IRT analyses and CAT simulations, an 8-item Friendship scale and 5-item Loneliness scale emerged from the Companionship domain. See Appendix A.

Social Distress Scales

Three factors were identified by the initial EFA (eigenvalues = 19.52, 2.47, and 1.66) conducted on the split sample (N=348). However, the 3-factor CFA displayed unacceptable model fit (CFI = 0.844, RMSEA = 0.106). Upon further investigation, the 8 items loading on factor 2 (Intrusiveness items) all displayed weak loadings (<.60) on the primary factor and/or high cross-loadings, and were thus eliminated. A 2-factor CFA run on the remaining items indicated acceptable model fit (CFI = 0.950, RMSEA = 0.106). Factor 1 included items representing aspects of perceived rejection, neglect or insensitivity from others, and was named Perceived Rejection. Factor 2 included items representing aspects of perceived hostility, ridicule or criticism from others; this factor was named Perceived Hostility. Following elimination of 2 items based on IRT analyses, CAT simulations were performed and top-ranking items selected to create 8-item Perceived Rejection and Perceived Hostility scales. See Appendix A.

Concurrent Validity

Psychometric properties of the six resulting adult NIH Toolbox SR scales can be found in Table 2, and correlations between the six NIH Toolbox SR scales and the three validation instruments (the ISEL, R-UCLA and NIS scales) are presented in Table 3. As would be expected, across the Toolbox SR scales, scales tapping similar SR concepts (ie, social support, companionship, and social distress) displayed correlations of relatively higher magnitude than those across the SR concepts (see highlighted cells, right side of table). The magnitude of these correlations (ranging from .73–.77) would indicate, however, that these scales assess related, but non-overlapping, constructs. In addition, cross-correlations of the NIH Toolbox SR scales with total scores from the social support (ISEL), loneliness (R-UCLS) and social distress (NIS) validity scales generally confirm the expected pattern of convergent validity (see highlighted cells, left side of table).

Table 2.

Psychometric properties for the adult Toolbox Social Relationship (SR) scales

	# of items	Mean (SD)Scale Score	Cronbach’s Alpha	CFI*	RMSEA*
Emotional Support	8	29.90 (8.59)	.969	.994	.112
Instrumental Support	8	29.24 (8.63)	.947	.966	.166
Friendship	8	26.53 (8.20)	.945	.985	.141
Loneliness	5	12.02 (5.31)	.939	.992	.267
Perceived Rejection	8	16.93 (6.47)	.932	.975	.142
Perceived Hostility	8	16.87 (6.92)	.941	.979	.101

Note: CFI = Complimentary Fit Index; RMSEA = Root Mean Square Error of Approximation

^*CFI and RMSEA determined for single factor model

Table 3.

Concurrent validity information for the adult Toolbox Social Relationship (SR) scales: Pearson correlation coefficients among Toolbox SR and validation scale scores

	Validation Instruments			Toolbox Social Relationship Scales
	ISEL	R-UCLA	NIS	ES	IS	FRI	LON	PR	PH
ISEL	***
R-UCLA	−.814	***
NIS	−.283	.382	***
Emotional Support (ES)	.790	−.800	−.264	***
Instrumental Support (IS)	.705	−.636	−.240	.754	***
Friendship (FRI)	.767	−.811	−.272	.733	.580	***
Loneliness (LON)	−.736	.854	.432	−.697	−.563	−.731	***
Perceived Rejection (PR)	−.580	.682	.606	−.573	−.491	−.573	.682	***
Perceived Hostility (PH)	−.359	.484	.657	−.367	−.285	−.400	.519	.770	***

Notes: Highlighted cells indicate hypothesized pattern of convergent validity (i.e., higher correlations) with specific validation instruments (left side of table) or between NIH Toolbox SR scales within the same conceptual domain (right side of table); ISEL=Interpersonal Support Evaluation List; R-UCLA=Revised UCLA Loneliness Scale; NIS=Negative Interaction Scale

^*All p values < 0.01; N=692

At first glance, patterns of discriminant validity across the NIH Toolbox scales and validity scales in Table 3 appear weaker, with higher magnitude correlations between the Toolbox Emotional Support scale and total R-UCLA (loneliness) score (−.80), and between the Toolbox Friendship scale and the total ISEL (social support) score (.77). However, additional data presented in Table 4 demonstrate that these relationships result from the multidimensional nature of the ISEL and R-UCLA instruments. As would be expected, the Toolbox Emotional Support scale is highly related to the ISEL Appraisal Support scale, but also with the R-UCLA Relational Connectedness scale. Similarly, the Toolbox Friendship scale is associated with the R-UCLA Collective Connectedness subscale, but also the ISEL Belonging subscale (see highlighted cells).

Table 4.

Post-hoc concurrent validity testing (Pearson correlation coefficient) between composite scores derived from the ISEL and R-UCLA Loneliness scales and relevant NIH Toolbox Social Relationship (SR) scales

	ISEL Social Support Composite Scores			R-UCLA Loneliness Composite Scores
	Appraisal	Tangible	Belonging	Isolation	Relational Connectedness	Collective Connectedness
Emotional Support	.810	.663	.633	−.642	.782	.683
Instrumental Support	.599	.707	.585	−.535	.578	.540
Friendship	.664	.645	.740	−.710	.696	.804
Loneliness	−.665	−.642	−.661	.824	−.683	−.691

Notes: Highlighted cells indicate hypothesized pattern of convergent validity (i.e., higher correlations) with specified components of the ISEL and R-UCLA validation instruments; ISEL=Interpersonal Support Evaluation List; R-UCLA=Revised UCLA Loneliness Scale

^*All p values < 0.001; N=692

Discussion

Despite evidence that social relationships influence physical health and well-being (Berkman & Glass, 2000; Suls & Wallston, 2003; Uchino, 2004), research in this area has been limited by a lack of brief yet precise measures of key social relationship concepts. The current report details the development and psychometric testing of the adult NIH Toolbox SR scales, a set of brief self-report scales designed to assess aspects of social relationships across community-based samples. Based on the extant literature, three concepts were selected for Toolbox assessment: social support, companionship, and social distress. While these scales were not intended to provide comprehensive coverage of all dimensions of social relationships, they represent a subset of key features of social relationship quality with potential relevance to the health psychologist.

As with much survey research, initial psychometric testing was conducted across reasonably large study cohorts, yet was limited in the number of scale items that could be tested. For this reason, significant effort was devoted to selecting, defining and clarifying the social relationship concepts included in the NIH Toolbox project. Scale items were closely reviewed and edited to reflect construct definitions, and to maximize item comprehension and relevance across a wide range of respondents. Finally, both classic and IRT-informed psychometric approaches were used to develop a set of brief, simple and unidimensional scales.

Use of IRT modeling to inform item selection for development of short-form scales carries a number of advantages over classical test theory approaches. IRT modeling allows one to evaluate the performance of individual items, and provides data regarding the amount of discriminating psychometric information that a single item can provide at all points along the severity scale of the latent construct of interest. Thus, use of IRT analyses and IRT-informed computerized adaptive testing (CAT) simulation data allow one to select items that provide maximal discrimination of underlying concepts of interest, and to develop short-form (5–8 item) scales that provide optimal information value across a range of individuals.

In the area of social support, brief scales were developed to assess aspects of Emotional and Instrumental Support. Notably, the Emotional Support scale includes items related to having someone available with whom to discuss problems or emotional difficulties, who may provide not only empathy and emotional support but also advice or suggestions about how to deal with life problems. While items related to receiving specific advice was originally categorized as reflecting informational (rather than emotional) support, results suggest that features of emotional support and advice-giving are likely to go hand-in-hand, or may not be distinguished by those receiving such support. In contrast, a separate scale was developed to assess aspects of Instrumental Support, or the availability of individuals who can provide functional aid in completing daily tasks such as cooking, cleaning, shopping, or transportation.

With respect to the concept of companionship, separate scales were developed to assess aspects of Friendship versus Loneliness. Separation across these dimensions is in line with previous factor analytic work with scales such as the R-UCLA (Hawkley, Browne, & Cacioppo, 2005), and may be driven not only by method variance (in which positively-worded friendship items and negatively-worded loneliness items tend to fall into separate factors), but also by genuine construct differentiation. Indeed, the existence of friends and companions versus feelings of loneliness may impact one’s physical and emotional health via differential social, psychological, physiological or health behavior mechanisms. Thus, assessing these concepts as independent dimensions may help to identify precise mechanisms by which these social relationship concepts impact health outcomes.

Relative to the growing research literatures regarding the positive impact of social support and the negative impact of loneliness on health outcomes, the impact of social distress on one’s physical and emotional health represents a neglected but equally important area of research. Thus, findings to support the feasibility of assessing aspects of Perceived Rejection and Perceived Hostility were encouraging. The Perceived Rejection scale includes items initially categorized as insensitivity/neglect, as well as items representing passive forms of rejection (i.e., people… avoid talking to me). In contrast, the Perceived Hostility scale includes items originally categorized as hostile/critical, as well as items describing active ridicule (people… tease me in a mean way). Notably, component items related to the construct of intrusiveness did not form a coherent factor. As with the assessment of intimacy, we would speculate that the experience of intrusiveness across social relationships may vary too widely to allow for brief yet global assessment of this SR dimension. It is also important to note that the validity measure of social distress utilized in this study was briefer and psychometrically weaker than the more established short-form scales used to assess social support and loneliness. Thus, additional research will be needed to support the construct validity of these novel social distress scales.

Parallel work conducted by the NIH Toolbox team has also evaluated similar social relationship scales for use in childhood (age 8–12) and adolescent (age 13–17) study samples. While inclusion of these pediatric data are beyond the scope of the current report, it is important to note that this work resulted in similar Emotional Support, Loneliness, Friendship, Perceived Rejection and Perceived Hostility self-report scales for use in respondents as young as age eight. These data support the notion that these key dimensions of social health are both relevant and measureable across the developmental spectrum. Future studies will, however, be needed to test the NIH Toolbox Social Relationship scales’ relevance, applicability and psychometric properties across special populations. For example, use of these scales in specific clinical samples, or in samples known to be experiencing significant relationship disruption, distress or isolation, will be helpful to determine both the sensitivity and specificity of these measures.

While a sizable literature exists on the assessment of social support and loneliness in adults, many extant scales espouse a multidimensional assessment approach, contributing to measurement imprecision. For example, factor analytic work by Hawkley et al (2005) suggests that the R-UCLA Loneliness scale assesses three separate concepts: Isolation, Relational Connectedness, and Collective Connectedness. Similarly, the ISEL (Cohen and Hoberman, 1983) was initially designed to assess multiple dimensions of social support including (but not limited to) Tangible Support, Appraisal Support, and Belonging. Review of item content indicates similarities across items loading on the ISEL Appraisal scale and the R-UCLA Relational Connectedness component (which both relate to the dimension of Emotional Support), and between items loading on the ISEL Belonging scale and the R-UCLA Collective Connectedness component (which both relate to the Friendship dimension). Such overlap likely contributes to high correlations observed between ISEL and R-UCLA total scores (r=−0.814 in the current sample). Thus, when utilized as indicators of loneliness or social support, these classic instruments may optimize conceptual breadth at the expense of measurement precision.

In contrast, the NIH Toolbox SR scales were deliberately engineered to provide brief assessments of key unidimensional social relationship constructs. This level of measurement precision may be crucial to support future research identifying neural circuits associated with responses to specific social cues, biological mechanisms linking social factors and health, and/or particular behavioral changes needed to improve social function. Indeed, developing precise measures of key dimensions of social relationship domains fits with the recent NIMH Research Domain Criteria (RDoC) project – which aspires to move future mental health research away from the use of multi-symptom diagnostic classification, in favor of identifying and assessing key underlying dimensions of emotional, cognitive or social function that become disrupted for some individuals. In this way, the current work may provide the precise, unidimensional social relationship scales needed to support future research in such emerging fields as social neuroscience, behavioral genetics and social behavior change.

Informed by IRT and classical test theory, the NIH Toolbox SR scales represent a promising and sophisticated attempt at measuring concepts of social support, companionship, and social distress across the lifespan. Future research is needed to further elaborate the construct validity, scale properties, and the potential utility of these scales for assessing social relationship quality within large-scale epidemiologic studies and clinical trials. However, the brevity and precision of these public-domain measures will contribute to their desirability and practical utility for use across a range of psychological, psychiatric and epidemiologic studies.

Appendix A.

Self-report scale items for adult Toolbox Social Relationship (SR) scales

Instruction Set for Social Support (Emotional / Instrumental Support) items: For the next set of questions, please read each statement and then decide how much each applies to you in the past month.  In the past month, please rate how often…

Emotional Support

1. I have someone who understands my problems

2. I have someone who will listen to me when I need to talk

3. I feel there are people I can talk to if I am upset

4. I have someone to talk with when I have a bad day

5. I have someone I trust to talk with about my problems

6. I have someone I trust to talk with about my feelings

7. I can get helpful advice from others when dealing with a problem

8. I have someone to turn to for suggestions about how to deal with a problem

Instrumental Support

1. Someone is around to make my meals if I am unable to do it myself

2. I have someone to take me shopping if I need it

3. I have someone to help me if I’m sick in bed

4. I have someone to pick up medicine for me if I need it

5. I have someone to take me to the doctor if I need it

6. There is someone around to help me if I need it

7. I can find someone to drive me places if I need it

8. I can get help cleaning up around my home if I need it

Instruction Set for Companionship (Friendship / Loneliness) items: For the next set of questions, please read each statement and then decide how much each applies to you in the past month.  In the past month, please rate how often…

Friendship

1. I get invited to go out and do things with other people

2. I have friends I get together with to relax

3. There are people around with whom to have fun

4. I can find a friend when I need one

5. I feel like I have lots of friends

6. I have friends who will have lunch with me when I want

7. I feel close to my friends

8. I feel like I’m part of a group of friends

Loneliness

1. I feel alone and apart from others

2. I feel left out

3. I feel that I am no longer close to anyone

4. I feel alone

5. I feel lonely

Instruction Set for Social Distress (Perceived Rejection / Hostility) items: For the next set of questions, please read each statement and then decide how much each applies to you in the past month.  In the past month, please rate how often people in your life…

Perceived Rejection

1. Don’t listen when I ask for help

2. Act like my problems aren’t that important

3. Let me down when I am counting on them

4. Act like they don’t have time for me

5. Act like they don’t want to hear about my problems

6. Act like they don’t care about me

7. Act like they can’t be bothered by me or my problems

8. Avoid talking to me

Perceived Hostility

1. Argue with me

2. Act in an angry way toward me

3. Criticize the way I do things

4. Yell at me

5. Get mad at me

6. Blame me when things go wrong

7. Act nasty to me

8. Tease me in a mean way