Development of CRIS: Measure of community reintegration of injured service members

Abstract

Identification and prevention of community reintegration problems of veterans is an important public health mandate. However, no veteran-specific measure exists. Study purposes were to (1) develop the Community Reintegration for Service Members (CRIS) measure and (2) test the validity and reliability of the measure. Formative research identified challenges in community reintegration postdeployment. The World Health Organization’s International Classification of Functioning, Disability and Health participation domain guided item-bank development. Items were refined through cognitive interviews and clinician consultation. Pilot studies with 126 veterans examined unidimensionality, internal consistency, reliability, and construct validity. Three unidimensional CRIS scales were developed. Working subjects had better CRIS scores then unemployed subjects. Subjects with posttraumatic stress disorder, substance abuse, or mental health problems had worse scores than subjects without these conditions. The correlations between the CRIS and the 36-Item Short Form Health Survey scales of role physical, role emotional, and social functioning were 0.44–0.80. CRIS has strong reliability, conceptual integrity, and construct validity.

INTRODUCTION

More than 1.5 million U.S. soldiers, sailors, and marines have been deployed in Operation Iraqi Freedom and Operation Enduring Freedom (OIF/OEF). Demobilization and the return home can be challenging, especially for injured veterans [1–2]. The ultimate goal of rehabilitative efforts is to help those injured adjust to life at home and in the community [3], which is also called community reintegration. Community reintegration is especially challenging for injured veterans because it may be complicated by the co-occurrence of physical injuries with postwar adjustment difficulties, such as posttraumatic stress disorder (PTSD), depression, substance abuse, and severe mental illness [1–2]. Community reintegration may be particularly problematic for OIF/OEF veterans, who have an unusually high prevalence of traumatic brain injury (TBI) [4], and PTSD [5–6].

Thus, it is important to assess community reintegration and, whenever possible, to intervene early to prevent long-term consequences for returning service members’ families and society. However, no community reintegration measure exists specifically for the veteran population. In fact, no single generic measurement tool has been developed that can be used in all populations [7]. Some existing measures are population-specific and have been developed, for example, for those with strokes, spinal cord injuries, or head injuries [3,8–9]. Furthermore, existing measures cover vastly different integration domains.

Recent revision of the World Health Organization’s International Classification of Functioning, Disability and Health (ICF) [10] offers an alternative method of defining and measuring the domains of community integration. According to the ICF, the domain of participation focuses on the person’s involvement in society (i.e., community integration). People are considered to have healthy participation if they take part in all life areas or life situations in which they wish to participate, in a manner or to the extent that is expected of an individual without restrictions in that culture of society [11]. The ICF taxonomy includes nine overall domains: learning and applying knowledge; general tasks and demands; communication; mobility; self-care; domestic life; interpersonal relationships; major life areas; and community, social, and civic life.

Many questionnaires that are used to assess participation in life were developed using narrower models of disability [12] and thus fail to cover all nine domains of participation as identified by the ICF. Our review found that very few measures contained at least one item pertaining to each of the nine domains and in many instances, single questions addressed more than one domain [13]. To date, no community integration measure has been developed that incorporates issues specific to injured service members. Even the most comprehensive existing measures do not have adequate coverage of subcategories relevant to the OIF/OEF veteran population based on a recently completed study that identified numerous areas of restriction in participation in OIF/OEF veterans [14].

Thus, we aimed to develop a new measure of community reintegration for injured service members: the Community Reintegration for Service Members (CRIS). The purposes of this study were to (1) design and develop a measure of community reintegration of injured service members using the results of our qualitative research on injured service members in conjunction with the ICF framework and (2) conduct preliminary tests of the measure’s validity and reliability.

CONCEPTUALIZING PARTICIPATION USING ICF

While the full description of the ICF taxonomy is beyond the scope of this article, a brief overview is provided here. The ICF model is divided into two components: the first covers functioning and disability, which includes four domains: (1) body function and (2) structure, (3) activities, and (4) participation. The second component of the model covers contextual factors including environment and personal factors. In the ICF graphic model, the body function and structure, activities, and participation domains are distinct from each other. However, in the ICF extensive system of taxonomy, only one coding structure exists for both activity and participation. Nevertheless, the two domains are conceptually distinct. According to the ICF, activities focus on the person’s individual functioning and are more likely to be performed alone [15]. In contrast, participation focuses on the person’s involvement in society (i.e., social functioning), and participation would more likely be performed with others.

Annex 3 of the ICF presents four options for differentiating between activities and participation. The first option is to exclusively designate some of the nine domains as activities and others as participation. The second option is to designate some domains as activities and others as participation—allowing partial overlap. The third option is to designate all broad categories as participation and all detailed categories as activities. In our approach to the measurement of community reintegration, we employed the fourth approach, which was to consider all codes as both activities and participation depending upon the content. Thus, as a general rule, we considered simple tasks and actions to be activities and complex functional tasks and actions to be participation (i.e., community reintegration). Questions pertaining to simple tasks were not included in the CRIS.

This approach is consistent with the finding of Jette et al., who demonstrated, using factor analysis, that activity and participation items from the Activity Measure for Post-Acute Care and the Participation Measure for Post-Acute Care (PM-PAC) loaded together within larger subdomains that roughly corresponded to ICF chapters [16].

METHODS

Overview

The study was conducted in three stages outlined as follows. In stage 1, challenges in community reintegration were identified through formative research [14]. Items related to each identified challenge were adapted from existing measures or written specifically for the CRIS. In stage 2, the CRIS item pool was tested in two pilot studies to evaluate structural validity and test-retest reliability. Finally, in stage 3, a subset of items with strong reliability and good fit were selected for inclusion in the fixed-form CRIS and internal consistency, test-retest reliability, and construct validity were examined.

Stage 1: Formative Research and Development of CRIS

A qualitative study was conducted with 14 injured veterans, 12 caregivers, and 14 clinicians. The study design and findings are reported elsewhere [14]. In summary, veterans and caregivers discussed challenges in daily life and role function. Clinicians discussed common challenges they had observed in OIF/OEF patients. Two investigators categorized the data using the ICF domains of activities and participation and a directed approach to content analysis [17]. Numerous community reintegration problem areas were identified and coded using the ICF taxonomy. These included challenges in acquiring complex skills; focusing attention; solving problems; reading; undertaking multiple tasks; carrying out daily routine; handling stress; producing communication; sustaining a conversation; moving around; using transportation; driving; self-care; maintaining one’s health; doing housework; caring for household objects; perceiving social cues; physical contact in relationships; forming relationships; regulating behavior; family and intimate relationships; acquiring, keeping, and terminating jobs; complex economic transactions; economic self-sufficiency; community life; recreation and leisure; socializing; political life; and citizenship [14].

Development of CRIS Item Pool

Item development was an iterative process that included item writing followed by cognitive testing, refinement, conceptual and psychometric evaluation, and further refinement [18]. Items forming the initial item pool were generated by adapting existing items and constructing new items. We began by reviewing existing community integration measures and behavioral outcome measures for TBI. We performed a content analysis of these measures, classifying every item into one or more of the nine ICF participation domains [19–20]. The content of each domain was further analyzed and coded into more precise subdomains by using the ICF taxonomy. Constructs and themes identified from the formative research phase were used to produce statements representing each of the dimensions and themes identified, which were then compared with the coded content of existing items. We developed new items when necessary.

In adapting existing items and writing new items, we created separate items for each of three dimensions of participation: frequency of participation, perceived limitation in participation, and satisfaction with participation [11]. Frequency of participation is considered a more objective measure [21] that does not consider individual perceptions, preferences, personal choices, and values. Because we believed that participation must also be understood in terms of consumer perceptions and experience, we developed separate scales to assess perceived limitation in participation and satisfaction with participation. Prior research shows that these dimensions (frequency, perceived limitations, and satisfaction) are only weakly and inconsistently correlated [22].

We phrased questions to facilitate comprehension, minimize recall bias, and enhance completion, which we felt was particularly important for service members who might have mild to moderate cognitive difficulties due to TBI. Thus, we worded questions to assess current life situation, with no comparison to life before injury or to other persons who were not injured or who had not been deployed, and framed questions to address participation within the last 2 weeks to minimize recall bias. Finally, we framed questions to minimize attribution to a specific health condition or event [23].

Content Validation and Cognitive Testing

Clinicians evaluated the content validity of proposed CRIS topic areas. Ten clinicians from Department of Veterans Affairs (VA) polytrauma and medical centers and the broader clinical community who had experience with injured service members were selected to participate in content validation. These clinicians were given descriptions of the ICF classification system along with written instructions to read over the list of proposed CRIS content areas and to gauge the importance of including each on a 7-point scale (1 = must include to 7 = should omit). We calculated average clinician ratings for each area and planned to omit content areas with average scores >4. No content areas were rated >4, and thus none was dropped.

Next, we asked seven additional clinicians to review the initial item bank and provide comments on language, content, and importance of each item on the same 7-point scale. Content raters included a polytrauma physician, two primary care physicians, a physical therapist, a recreational therapist, a psychologist, and an occupational therapist. Several questions with scores >4 were deleted and several questions were reworded or new items written based upon these clinicians’ suggestions. Next, we conducted cognitive-based interviews with seven OIF/OEF veterans to provide feedback about the questions and response formats and further refinements were made based upon cognitive testing [24–25]. The initial CRIS item bank consisted of items divided into 30 scales: a 97-item Extent of Participation scale, a 116-item Perceived Limitation in Participation scale, and an 83-item Participation Satisfaction scale. Each item used 7-point Likert-type scales.

Stage 2: Pilot Studies

Two pilot studies of the CRIS item bank were conducted. Both studies examined structural validity of the CRIS scales to examine unidimensionality of the hypothesized domains, evaluate internal consistency of the scale items, and examine construct validity using known group analysis. The second pilot study examined test-retest reliability of the scales.

Pilot Study 1

Sample and Design

Pilot 1 employed a convenience sample of 50 veterans recruited from the primary care service of the Providence VA Medical Center (PVAMC) (Table 1), including both primary care homeless veteran clinics and usual primary care clinics.

Table 1.

Subject characteristics: Pilot studies.

Variable	Pilot 1 (N = 50)		Pilot 2 (N = 76)
	n	%	n	%
Age (yr)
18–25	0	0	5	7
26–35	4	8	9	11
36–45	7	14	17	22
46–55	17	34	30	39
56–65	16	32	15	20
65+	6	12	0	0
Race
White	43	86	55	73
Black	3	6	9	12
Other	4	8	12	16
Male	44	88	61	80
Marital Status
Single	16	32	16	21
Married	15	30	21	28
Engaged	2	4	2	3
Long-Term Relationship	9	18	11	15
Divorced or Separated	6	12	23	30
Widowed	2	4	3	4
War
OIF/OEF	6	12	24	32
Vietnam	24	48	25	33
Other	17	40	27	35
PTSD	11	23	24	32
Depression	16	33	34	45
Substance Abuse	18	37	31	42
Any Mental Health Dx	23	46	54	59
Employment
Working for Pay	14	28	36	47
Unemployed	25	50	38	50
Retired	9	18	2	3
Student	1	2
Volunteer	1	2
Education
<High School	2	4	0	0
High School or GED	16	33	19	25
Some College	24	48	32	42
College	6	12	13	17
Postgraduate	1	2	5	7
Missing	1	2	0	0
Income ($)
<15,000	15	30	4	5
15,000–24,999	9	18	14	18
25,000–34,999	12	24	9	12
35,000–50,000	6	12	16	21
>50,000	7	14	30	38
Missing	1	2	3	4
Residence Past 2 Weeks
Apartment	13	26	34	45
House or Condominium	26	52	39	51
Group Home/Shelter/Other	11	22	3	4

Dx = diagnosis, GED = General Educational Development, OIF/OEF = Operation Iraqi Freedom and Operation Enduring Freedom, PTSD = posttraumatic stress disorder.

Approach

We conducted preliminary analyses of unidimensionality of CRIS scales using WinSteps (Beaverton, Oregon) [26] and selecting items with the smallest percentage of missing data. Questions related to parenting and employment were omitted because of sample size limitations. We ran separate Rasch models within each hypothesized dimension (Extent of Participation, Perceived Limitation in Participation, Participation Satisfaction), examined the item-person map, and then checked the weighted fit statistics or mean square infit statistic and Rasch residual factor analysis to test the model fit and unidimensionality.

The Rasch model is a probability model that describes the person’s response as a function of person and item parameters. The person parameter is called the latent variable and represents a person’s underlying ability. In a Rasch model, the item parameter only has the difficulty parameters or threshold parameters and the discrimination parameter is set as 1 [27]. WinSteps is a statistical program that can be used to estimate the item and person parameters of a Rasch model. It is based on joint maximum likelihood estimation [26]. An item-person map provides a general picture of how good the match is between the scale and the sample. An infit statistic >1.4 represents noise in the data and indicates that the item does not belong to the unidimensional construct [26]. We examined the scale plot of the item measures and their corresponding factor loadings to determine whether a systemic pattern between positive and negative loading existed in terms of the item content. The Rasch residual factor analysis extracts the common factor underlying the items that explains the most residual variance. If this factor is only random noise, then the residuals have no meaningful structure [26]. Next, we examined the alphas for each of the CRIS dimensions identified in the analysis.

To assess CRIS construct validity, we examined differences in CRIS scores by known groups’ validation. “Known group validity” refers to the comparison of mean difference across known categories. The mean of different categories was calculated based on the person scores estimated from the Rasch model. The sample was classified into three known groups: (1) engagement in productive activity, (2) presence of PTSD, and (3) presence of depression. We hypothesized that veterans who were (1) working or volunteering and did not have (2) PTSD or (3) depression would be better integrated into the community and thus would have higher (i.e., better) CRIS scores.

Productive activity was a 3-level variable categorized as engaged in productive activity (which included those working or volunteering), retired, and not engaged in productive activity. PTSD and depression were dichotomous variables based on documented diagnoses in the medical record. Four separate analyses of variance (ANOVAs) were performed using the independent variable of productive activity and dependent variables of CRIS scales (description follows).

Results

Descriptive analyses of the preliminary Rasch modeling identified a distinct Extent of Participation dimension; the item-person map was acceptable, the scale covered the entire sample, and no gap in the scale was noted. Of the 60 items, 8 were misfit. A systemic pattern emerged in the residual matrix. Within this analysis, we identified two subscales in the Extent of Participation domain that we interpreted as an Emotional/Cognitive subdomain and a Social/Interpersonal subdomain. Rasch analyses were also used to examine the Perceived Limitation in Participation and the Participation Satisfaction domains; however, the underlying structure was less distinct and the scales needed additional items at the highest and lowest levels of participation within this sample. In the Participation Satisfaction domain, 0 of 71 items were misfit, while in the Perceived Limitation in Participation domain, 16 out of 103 items were misfit.

The percentage of variance in the data explained by each of the three Rasch models (i.e., Extent of Participation, Perceived Limitation in Participation, Participation Satisfaction domains) was 64.3, 59.7, and 71.3 percent, respectively. Alphas of scales identified in the item response theory (IRT) analysis ranged from 0.82 to 0.98. We calculated scores for each of the four scales identified in the preliminary IRT analysis as the mean score of all completed questions within the scale multiplied by the number 10. Thus the range of scores for each of the scales was 10 to 70, with higher scores indicating better integration.

As hypothesized, descriptive statistics of scale scores showed differences in CRIS scale scores between groups (results not shown). Subjects who were working had higher scores on all four scales than those who were retired or not working. Results of the ANOVA showed lower (i.e., worse) scores on three of the four scales for veterans with PTSD compared with veterans without PTSD. Results of the ANOVA on depression indicated comparable scores for all CRIS scales, except for Participation Satisfaction, which was significantly lower for veterans with depression (p = 0.03).

We used the results of pilot 1 to refine the CRIS measure, revising all misfit items and adding additional items to address the higher and lower ends of participation of the Perceived Limitation in Participation and the Participation Satisfaction domains. All new items were cognitively tested in a sample of six veterans recruited from PVAMC’s primary care clinics. Following refinement of the new items, we conducted a second pilot study of psychometrics of the CRIS.

Pilot Study 2

Sample and Design

A convenience sample of 76 subjects was recruited from the PVAMC’s primary care clinics. Efforts were made to recruit into the sample veterans who were parents of young children and veterans who were working. Subject characteristics are shown in Table 1. During the first visit, each subject was administered the CRIS items and demographic questions. During the second visit, CRIS items as well as the Craig Handicap Assessment and Reporting Technique (CHART) and the 36-Item Short Form Health Survey (SF-36) were readministered to the subjects. Sixty-six subjects returned for a follow-up visit within 1 week.

Approach

Structural validity analyses were conducted using data from the first visit. We performed preliminary Rasch analyses following the methods described in pilot 1. To assess the preliminary construct validity of the revised measure, we examined differences in scores by known groups’ validation following the methods described in pilot 1. Mean ± standard deviation scores for the overall sample were as follows: Extent of Participation 50.2 ± 8.7 (range 28.0–64.0), Perceived Limitation in Participation 50.1 ± 10.2 (range 28.1–69.4), and Participation Satisfaction 50.0 ± 10.3 (range 23.1–68.9). The sample was classified into four known groups: (1) employment status, (2) diagnosis of PTSD, (3) diagnosis of substance abuse, and (4) diagnosis of any mental health problem (inclusive of PTSD, depression, substance abuse, and other diagnoses). Because the sample for pilot 2 was younger with fewer participants who were retired than the sample in pilot 1, we classified employment status as either working (which included those working full-time or part-time or in work training) or not working (because of disability, unemployment, or retirement). We classified participants based upon the presence or absence of any mental health problem based on documented diagnoses in the medical record. We hypothesized that (1) veterans who were working and (2) those without PTSD, (3) substance abuse, or (4) any mental health problem would be better integrated into the community and thus would have higher (i.e., better) CRIS scores.

Three separate general linear models (GLMs), controlling for potential confounding by age, were performed using the independent variable of employment status and dependent variables of CRIS (1) Extent of Participation scale, (2) Perceived Limitation in Participation scale, and (3) Participation Satisfaction scale. Similar sets of GLMs were performed using the independent variables of PTSD, substance abuse, and any mental health problem.

Reliability was examined by calculating the intraclass correlation coefficient (ICC) of each scale. ICCs were calculated as the Shrout & Fleiss (type 2,1) ICC, which is generally denoted by ICC (2,1) [28]. ICC (2,1) is a two-way mixed-effects single-measure reliability measure in which the target is random effect, the number of measurements on each target is fixed effect, and the unit of analysis is the individual measurement instead of the mean of measurements.

Results

These structural analyses revealed three distinct, unidimensional subdomains of community participation. The results showed that unidimensionality of the Extent of Participation scale improved in the second version of the CRIS item bank, with only one dimension identified, in contrast to the findings from pilot 1, which identified two subdomains in the Extent of Participation scale. The percentage of variance in the data explained by each of the three Rasch models (i.e., Extent of Participation, Perceived Limitation in Participation, Participation Satisfaction subdomains) was 53.2, 85.2, and 73.3 percent, respectively. In the Extent of Participation scale, the distribution of items across the range of community participation (referred to as the item-person map) was acceptable, the scale covered the entire sample, and no gap in the scale was noted. In Rasch analyses of the Perceived Limitation in Participation scale, a gap at the highest levels of participation within this sample was revealed, indicating a need for additional items in the item bank. These results show that the gap at the lower level of Perceived Limitation in Participation (identified in the pilot 1 analysis) was closed in the second version of CRIS. In the Participation Satisfaction scale, gaps were identified at the highest and lowest levels of participation within this sample.

In the Extent of Participation scale, 3 items were misfit; in the Perceived Limitation in Participation scale, 20 items were misfit; and in the Participation Satisfaction scale, 14 items were misfit. Misfit items were removed and the alpha of each scale was calculated. The final scales included 95 items in the Extent of Participation scale, 107 items in the Perceived Limitation in Participation scale, and 85 items in the Participation Satisfaction scale. Scale alphas showed excellent internal consistency. Internal consistency reliability coefficients were 0.92 for Extent of Participation, 0.94 for Perceived Limitation in Participation, and 0.93 for Participation Satisfaction.

GLM results showed statistically significant differences in scores between groups in the hypothesized direction. Subjects who were working had higher scores on all scales than those who were not working. Results of the GLMs showed lower scores on all scales for veterans with PTSD, substance abuse history, or mental health problems than veterans without those diagnoses.

The ICC for the Extent of Participation reliability was 0.92 (95% confidence interval [CI] 0.87 to 0.95). The ICC for the Perceived Limitation in Participation scale was 0.94 (95% CI 0.91 to 0.96), while the ICC for the Participation Satisfaction scale was 0.93 (95% CI 0.89 to 0.96).

Stage 3: Fixed-Form CRIS Scales

Design

Our objective was to develop three separate comprehensive scales, one for frequency of participation, one for perceived limitation in participation, and one for satisfaction with participation, that could be administered in approximately 15 minutes each.

Approach

We chose items for inclusion based on comprehensiveness of content and item reliability, keeping in mind the preliminary assessment of item difficulty as calculated by the IRT analysis. Thus, we included one or more items that addressed each content area identified in our formative research and attempted to obtain items covering the entire range of item difficulty. We included only items with ICCs of 0.6 or higher. After making initial selections, we examined item-to-total correlations within each scale and replaced any items with correlations lower than 0.2 with alternative items that covered the same content area. Linear regression models (with fixed-form scores as independent variables) were used to estimate the proportion of variance (R²) explained in full scales. To examine the construct validity of the fixed-form scales, we repeated the GLMs performed in pilot 2, this time using scores of fixed-form scales as dependent variables. Thus, we performed separate GLMs, controlling for potential confounding by age, using the independent variables of employment status, PTSD, substance abuse, and any mental health problems for each of the CRIS scales.

We examined convergent and discriminant validity of the CRIS fixed-form scales by performing pairwise correlations of the CRIS scales/measures with other items, which in theory should be highly related, as well as with measures, which in theory should not be strongly related. We examined correlations between CRIS sub-scales and two subscales of the CHART [8]: the occupation scale and the social integration scale; and four scales from the SF-36: physical functioning, role physical, role emotional, and social functioning. The 7-item CHART occupation subscale assesses number of hours of participation in occupational activities. The 6-item CHART social integration subscale assesses type of living arrangements; presence or absence of romantic involvement; and amount of contact with friends, associates, and strangers. The physical function scale is a 10-item sub-scale of the SF-36 that measures difficulty with performance of physical activities [29]. The 3-item SF-36 role emotional subscale measures difficulty with role function in work or daily activities attributable to mental health problems. Finally, the 2-item SF-36 social functioning subscale measures interference with social activities related to physical and emotional problems [29].

RESULTS

Alpha coefficients of the three fixed-form scales were Extent of Participation 0.91, Perceived Limitation in Participation 0.93, and Participation Satisfaction 0.97. Fixed forms, which employed approximately half the number of items of the full scale, were able to predict 0.97–0.98 of the variance within the full scales. A summary of CRIS items classified by domain and subcategory of activities and participation taxonomy is shown in Table 2. The CRIS scales are included in the Appendix (available online only).

Table 2.

Summary of number of items in each content area by International Classification of Functioning, Disability and Health (ICF) coding classification.

ICF Chapter and Subcategory	Extent of Participation	Perceived Limitation in Participation	Participation Satisfaction
Learning and Applying Knowledge	7	7	7
d160–d179 Applying Knowledge
General Tasks and Demands	7	6	6
d220 Undertaking Multiple Tasks
d230 Carrying Out Daily Routine
d240 Handling Stress and Other Psychological Demands
Communication	2	3	2
d330–d349 Communication Producing
d350–d369 Conversation and Using Communication Devices
and Techniques
Mobility	5	5	4
d460 Moving Around in Different Locations
d470–dd489 Moving Around Using Transportation
Self-Care	7	6	6
d500 General Self-Care
d570 Looking After One’s Health
Domestic Life	2	3	4
d630–d649 Household Tasks
d650–d669 Caring for Household Objects and Assisting Others
Interpersonal Relationships	6	10	9
d710 Basic Interpersonal Interactions
d720 Complex Interpersonal Interactions
d730–d779 Particular Interpersonal Relationships
Major Life Areas	4	8	5
d840–d859 Work and Employment
d860–d869 Economic Life
Community Social and Civic Life	7	5	4
d910 Community Life
d920 Recreation and Leisure
d950 Political Life and Citizenship
Total Number of Items	47	53	47

As hypothesized, results of the GLMs show differences in CRIS scale scores between groups (Table 3). After controlling for age, subjects who were working had higher scores on all scales than those who were not working. Subjects with PTSD, substance abuse history, or mental health problems had lower scores than veterans without those diagnoses.

Table 3.

Descriptive statistics and differences by known groups of Community Reintegration for Service Members (CRIS) scale scores.^*

CRIS Scale	Extent of Participation		Perceived Limitation in Participation		Participation Satisfaction
	Mean ± SD	p-Value	Mean ± SD	p-Value	Mean ± SD	p-Value
Employment		0.012		0.004		0.02
Working	52.0 ± 8.4		52.7 ± 9.7		51.4 ± 9.3
Not Working	49.5 ± 9.3		49.5 ± 1.0		49.8 ± 1.1
Diagnosis of PTSD		<0.001		0.001		0.001
Yes	44.1 ± 7.6		44.0 ± 8.7		44.1 ± 10.2
No	53.1 ± 7.9		53.0 ± 9.6		52.7 ± 9.3
Diagnosis of Substance Abuse		<0.001		0.007		0.04
Yes	47.0 ± 9.0		46.5 ± 10		47.2 ± 1.1
No	52.6 ± 8.1		52.9 ± 9.4		52.1 ± 9.6
Any Mental Health Diagnosis		0.001		0.001		0.001
Yes	48.0 ± 9.6		47.5 ± 9.8		47.8 ± 10.2
No	51.4 ± 9.2		52.0 ± 10.4		51.6 ± 10.6

^*Results of general linear models controlling for age.

PTSD = posttraumatic stress disorder, SD = standard deviation.

Results of the pairwise correlation analysis (Table 4) show that CRIS scales were strongly correlated with the SF-36 scales of role physical, role emotional, and social functioning. In contrast, CRIS scales were weakly correlated with the CHART occupation scale and not significantly correlated with the CHART social integration scale. CRIS scales were weakly to moderately correlated with the SF-36 physical function scale.

Table 4.

Correlations between Community Reintegration for Service Members scales and other measures.

Measure	Extent of Participation		Perceived Limitation in Participation		Participation Satisfaction
	R	p-Value	R	p-Value	R	p-Value
Occupation (CHART)	0.27	0.02	0.25	0.04	0.23	0.06
Social Integration (CHART)	0.096	0.44	−0.12	0.33	−0.20	0.10
Physical Function (SF-36)	0.37	<0.01	0.38	<0.01	0.21	0.08
Role Physical (SF-36)	0.62	<0.001	0.62	<0.001	0.44	<0.001
Role Emotional (SF-36)	0.80	<0.001	0.79	<0.001	0.67	<0.001
Social Function (SF-36)	0.75	<0.001	0.78	<0.001	0.65	<0.001

CHART = Craig Handicap Assessment and Reporting Technique, SF-36 = 36-Item Short Form Health Survey.

DISCUSSION

The results of our analyses demonstrate that the CRIS instrument is a comprehensive measure with conceptual integrity; excellent reliability; strong content; and construct, convergent, and discriminant validity. The validity of the dimensions is based on face validity, item fit, and the Rasch residual factor analysis. These results showed that most of the items fit the data across two pilot studies: more than 50 percent of the variance could be explained by the Rasch model in two pilot studies and the residual pattern generally supported the unidimensionality within each proposed domain.

The construct validity of the CRIS measure is supported by the analysis of known group differences and by the concurrent and discriminant validity analyses. As expected, we observed moderate to strong correlations between CRIS scales and three SF-36 scales. Each of these scales aims to measure the constructs of role and social functioning. We observed weaker to no correlations been CHART subscales and CRIS scores, because CHART scales measure only quantity of occupational engagement, i.e., hours of work or productive activity and number of friends or business associates. In contrast, CRIS and SF-36 scales ask about the frequency of restrictions in participation, perceived limitations in participation, and participation satisfaction. The stronger correlation between the CRIS Extent of Participation scale and the SF-36 versus the CHART suggests that this scale is a more subjective measure of frequency of participation and difficulties with participation (similar to SF-36 scales) than the CHART. The weak to moderate correlations between the SF-36 physical function scale and CRIS scales make sense given that a person’s participation in role function is influenced by activity limitations as well as the environment and other personal factors.

We believe that further development, validation, and use of the CRIS is important. Numerous reintegration problems in Gulf War and OIF/OEF veterans have been reported, including marital difficulties, financial difficulties, problems with alcohol or substance abuse, medical problems, behavioral problems such as depression or anxiety [2], homelessness [30], and motor vehicle accidents [31]. To date, no systematic efforts have estimated the scope of these problems. At present, neither the VA nor Department of Defense electronic medical records contain standardized data elements related to functional health and community reintegration. Use of the CRIS would provide a method for comprehensive, standardized assessment and monitoring of community reintegration outcomes of these vulnerable veterans. Unlike the PM-PAC or other generic measures of participation, the CRIS is specially designed to address community integration issues from a service person’s perspective. In future studies, researchers could compare the CRIS to generic measures of participation, such as the PM-PAC, to determine whether value is added with a service person-specific measure.

Limitations

One limitation of our study is that it was not possible to do exploratory and confirmatory factor analysis because of the small sample size. However, we believe that our relatively small sample size was adequate to yield stable parameter estimates within each major domain because of the few parameters in the Rasch model. Linacre has suggested that a sample of 50 well-targeted examinees is a conservative estimate of the sample size required to obtaining useful estimates; thus our pilot sizes, while not ideal, were adequate for generating estimates in a well-designed, focused pilot study [32]. From the item parameter recovery point of view, evidence suggests that increasing the number of items to be analyzed has little effect on the item parameter recovery but increasing the number of categories will increase the error variance of the parameter estimates [33]. The sample size, therefore, was reasonable for a Rasch model pilot study. We considered using a two-parameter logistic model to assess discrimination and fit of items: the use of this type of model would have required a larger sample size than was available to us in this pilot work.

Enhanced clinical information systems are a key component of improving care delivery for patients with chronic and complex conditions. Routine assessment of community reintegration would enhance patient assessment and targeting of referrals to services such as mental health, social services, and benefits as well as drive interventions that address underlying factors related to poor community reintegration. However, the use of fixed-form assessment methodology for a comprehensive reintegration instrument like the CRIS presents common problems that may limit its ultimate utility.

Future Research Directions

The fundamental problem encountered with short-form instruments is the occurrence of frequent floor and ceiling effects, in which large numbers of individuals score at either the top or the bottom of the range of possible scores. These effects severely reduce measurement precision [34]. The CRIS was developed as a more comprehensive and lengthy instrument (with 147 items across three basic subscales) to minimize concerns about inadequate measurement precision and inadequate coverage of important participation domains. While achieving excellent psychometric properties, as demonstrated in these analyses, the length of the CRIS may limit its utility in the clinical environment.

One promising solution to measurement problems of traditional fixed-form instruments is offered through the combined application of computer-adapted testing (CAT) [35] and IRT [36–37]. CAT methodology uses a computer interface for the patient that is tailored to the unique ability level of the patient on a specific outcome dimension. Administering outcome items that are either too easy or too hard provides little information. An adaptive test first asks questions in the middle of the ability range and then directs questions to the level based on the patient’s responses, without asking unnecessary questions. This allows for administration of fewer items while gaining precise information regarding an individual’s placement along a continuum of community integration. The strategy of matching items to respondents has been used to achieve short and precise educational and psychological tests for decades. CAT results from the use of a simple form of artificial intelligence that selects questions tailored to the test-taker, shortens or lengthens the test to achieve the desired precision, scores everyone on the standard metric so that results can be compared, and displays results instantly. In practice, this approach minimizes the number of items that are administered to an individual and obtains an estimate of participation in any particular content area.

CAT applications require a large set of items in any one outcome domain (called item banks); items that consistently scale along a dimension of low to high proficiency; and rules guiding starting, stopping, and scoring procedures. The current fixed-form CRIS can provide the foundation for an item pool that could be used to develop a CAT version of this instrument. A significant challenge in developing CAT instruments is the need for large, representative data sets that are used to establish item and response characteristic curves in item pools that are needed for the complex modeling in the CAT programs [35].

CONCLUSIONS

The careful use of CAT-based outcome assessment applied to community reintegration holds considerable promise to make outcomes assessments briefer and less burdensome to patients and thus more acceptable for use in busy clinical and research settings. CAT methodology will provide data that is applicable across different clinical settings and applications, more efficient and less costly to administer, and where needed, more precise than conventional assessment approaches. Thus, they may have considerable promise as a vehicle for advancing community reintegration assessment while avoiding the pitfalls of traditional methodology. The CRIS is a comprehensive instrument with sound psychometric properties that can be used to measure community reintegration of veterans. To reduce respondent burden, we recommend further development of the CRIS through the combined application of CAT and IRT.

Abbreviations

ANOVA

analysis of variance

CAT

computer-adapted testing

CHART

Craig Handicap Assessment and Reporting Technique

CI

confidence interval

CRIS

Community Reintegration for Service Members

GLM

general linear model

ICC

intraclass correlation coefficient

ICF

International Classification of Functioning, Disability and Health

IRT

item response theory

OIF/OEF

Operation Iraqi Freedom and Operation Enduring Freedom

PM-PAC

Participation Measure for Post-Acute Care

PTSD

posttraumatic stress disorder

PVAMC

Providence VA Medical Center

SF-36

36-Item Short Form Health Survey

TBI

traumatic brain injury

VA

Department of Veterans Affairs