Environmental Factors Item Development for Persons With Stroke, Traumatic Brain Injury, and Spinal Cord Injury

Abstract

Objectives

To describe methods used in operationalizing environmental factors; to describe the results of a research project to develop measures of environmental factors that affect participation; and to define an initial item set of facilitators and barriers to participation after stroke, traumatic brain injury, and spinal cord injury.

Design

Instrument development included an extensive literature review, item classification and selection, item writing, and cognitive testing following the approach of the Patient-Reported Outcomes Measurement Information System.

Setting

Community.

Participants

Content area and outcome measurement experts (n=10) contributed to instrument development; individuals (n=200) with the target conditions participated in focus groups and in cognitive testing (n=15).

Interventions

None.

Main Outcome Measures

Environmental factor items were categorized in 6 domains: assistive technology; built and natural environment; social environment; services, systems, and policies; access to information and technology; and economic quality of life.

Results

We binned 2273 items across the 6 domains, winnowed this pool to 291 items for cognitive testing, and recommended 274 items for pilot data collection.

Conclusions

Five of the 6 domains correspond closely to the International Classification of Functioning, Disability and Health taxonomy of environmental factors; the sixth domain, economic quality of life, reflects an important construct that reflects financial resources that affect participation. Testing with a new and larger sample is underway to evaluate reliability, validity, and sensitivity.

The General Assembly of the United Nations’ has identified participation as a primary and highly valued goal of rehabilitation, health care, and social services for people with disabilities.¹ People with disabilities and their advocates emphasize that the physical, social, political, and economic aspects of the environment are important influences on health and participation.² The World Health Organization’s International Classification of Functioning, Disability and Health (ICF)³ has become the de facto taxonomy by which to describe the precursors and consequences of disability. The ICF identifies environmental factors into the following 5 categories: products and technology; natural environment; support and relations; attitudes; and services, systems, and policies. In spite of its ascendency, the ICF has been criticized for failing to provide a model of the disablement/enablement process.⁴ The ICF provides no less than 3 different coding conventions for its 5 environmental factor chapters, leaving instrument developers with considerable latitude in how to operationalize environmental factors.

Many measures of the physical and social environment lack a strong theoretical foundation^5,6 and fail to clarify the dynamic interaction between participation and environment.⁷ Mallinson and Hammel⁸ emphasize the need for a transactional measurement approach that describes activity and participation in the context of the environment, including physical, social, and attitudinal supports; barriers to task and role performance; and social and community participation.

In spite of these conceptual and measurement challenges, the last decade has seen increased interest in the development of measures of the environment, and there are several high-quality reviews of these instruments in the rehabilitation literature.^5,9 For example, Whiteneck,⁵ Noreau,⁹ and colleagues describe methodologic challenges in operationalizing environmental factors. There is, however, no consensus on which instruments are best for describing participation-environment interactions. Empirical evidence of the impact of the environment on participation is inconsistent.^9,10

Many contemporary measures of the environment use the classical test theory and combine multiple aspects of the environment into a single scale or report descriptive, item-level information only; these issues are reviewed in the first article of this special issue.¹¹ Given that participation occurs across multifaceted environments, instrument development must strike a balance between measurement precision and respondent burden.⁵ Rather than try to capture the environment in its entirety (which would be very difficult to do), we should focus on assessing aspects of the environment that either are amenable to change or are expected to affect specific outcomes in targeted communities. Although it remains unclear what the most appropriate measurement models are for assessing environmental factors and their impact on participation, it is evident that we need additional research and approaches to refine the construct of environmental facilitators and barriers to participation.

Contemporary psychometric methods and the application of item response theory provide a viable approach to measurement of environmental factors. In this study, we sought to apply a fresh approach to the measurement of environmental factors using the mixed-methods instrument development process championed by the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative.¹² We sought to focus on environmental factors that people with disabilities perceive as facilitating participation or acting as barriers. We purposely focused on the subjective experiences of environmental factors, which are best captured by self-report, rather than objective environmental descriptions. These issues are summarized in the second article of this special issue.¹³ Therefore, the objectives of this article are (1) to describe the methods used to operationalize environmental factors; (2) to describe the results of a research project to develop measures of environmental factors that affect participation; and (3) to describe an initial item set of facilitators and barriers to participation. The goal of the larger project of which this study is a component is to develop, test, and evaluate measures of environmental factors and their influence on participation for persons with stroke, traumatic brain injury (TBI), and spinal cord injury (SCI). A validated set of environmental factor instruments will allow us to determine to what extent they influence social health and participation.

Methods

We used a mixed-methods approach to constructing and evaluating items. The major tasks, which are subsequently described, entailed concept elicitation, item classification and creation, and item refinement. These tasks and procedures follow the standardsestablished by the PROMIS Network.¹² A domain chair and cochair oversaw the development of each item set. We obtained institutional review board approval for this project prior to the start of this project.

Phase I: Concept elicitation, qualitative focus group analysis, and conceptual modeling

As part of our ongoing research on participation, we had access to verbatim transcripts from 38 focus groups involving >200 people with diverse disabilities across different sites and regions.^14-16 The samples included people with stroke, TBI, SCI, and other disabilities. We analyzed these data to create participation and quality of life assessments (eg, community participation indicators, SCI quality of life/TBI quality of life measurement systems).^16-18 In the focus groups, participants stressed the importance of environmental factors influencing participation; the depth and detail of environmental factor information allowed us to reanalyze the data for the current project. We used a grounded theory approach and constant comparative approach to coding and interpreting.¹⁹ This analysis yielded rich environmental factor themes, detailed examples, and relevant quotes to highlight specific factors across the groups. We categorized the feedback in 8 distinct environmental factor domains identified as important by people with disabilities. The second column of table 1 lists the domains. Some of these domains directly correspond to ICF typology shown in column 1 (eg, systems, services); others focus on specific components of ICF environmental factor chapters (eg, transportation, assistive technology [AT]), and one represents an issue that is an infrequent focus of research, economic quality of life. We grouped social supports and societal attitudes together as a social environmental domain.

Table 1.

Comparison of ICF and project-specific conceptual domains

ICF Taxonomy	Initial Framework	Binned Items	Winnowed Items	Revised	Cognitive Interviews Item Count	Pilot Test Item Count
Products and technology	AT	178	104	AT	15*	14*
Natural environment   and human made changes	Natural environment	79	37	Built and natural environment	36	36
	Built environment	605	51
Support and relations  Attitudes	Social supports and  attitudes	710	91	Social environment	90	82
Services, systems, and policies	Services, systems,  policies	411	35	Services, systems, policies	76	75
	Transportation	136	28
	Access to information  and technology	112	37	Access to information and  technology	32	25
	Economic, financial	42	35	Economic quality of life	37	37
Total		2273	392		291	274

^*AT items are repeated for mobility device, activities of daily living equipment, and devices used to interact with people and environments. There was also 1 item developed to assess barriers to these devices.

Phase II: Item classification and creation

We binned items by assigning them to domains using expert opinion. Therefore, we winnowed items, removing items with similar content, using expert opinion. We created a working definition of each thematic code based on focus group data, review of legacy assessments, and environmental factor research. We used this guide to categorize and generate sample items within each domain.

Phase III: Item refinement

Cognitive interviews

Following the PROMIS’ qualitative item review process,¹² we conducted cognitive interviews with community-dwelling people with disabilities to ensure comprehension of items. Participants completed a subset of up to 50 items. We asked participants to explain what the question meant in their own words and to describe how they arrived at the answer. Interviewers probed to clarify the participants’ interpretation of specific words and phrases. Each interview lasted 20 to 40 minutes. Most participants volunteered to complete 2 to 3 interviews, requiring 1 to 4 hours in total. Each item was reviewed by at least 4 and up to 9 participants who had different disabilities. The project team worked to ensure that members of each diagnostic group reviewed items in each subset to minimize sampling bias. Participants received an honorarium and accommodations as necessary. Accommodations included increasing time, discussing confusion regarding item content, and providing breaks as needed.

Interviewers took verbatim notes during the interviews and reviewed them with the team. We coded responses, and the domain chair and cochairs reviewed them. The research team reviewed and discussed problematic items. Revised items underwent a second round of cognitive testing with an additional 3 participants. Once the research team reached consensus on the suitability of the revised items, we deemed the items ready for pilot testing.

Reading level analysis

Patient literacy is a well-documented challenge for use of self-administered questionnaires.^15,20-22 Although we designed the items to be self-administered, it is essential to verify that respondents comprehend their wording and the meanings are clear. High literacy expectations, complex terms, and ambiguous items can create comprehension problems and impair test validity. We assessed the reading level of each item using the Flesch-Kincaid and Lexile Framework²³ and reworded items that were above a fifth grade level. Maintaining a fifth grade reading level maximizes the scale’s accessibility to participants with low reading skills.

Translatability review

In anticipation of future translation of items to Spanish, we followed the PROMIS guidelines for translation and cultural adaptation.²⁴ This process involves forward and backward translations of items, response options, and instructions. We followed an iterative process of forward and back translation with bilingual expert review.

Results

Literature review

We searched PsychLit, PubMed, and Cumulative Index to Nursing and Allied Health Literature using the terms environment, social environment, and environmental factors as keywords and identified articles that also contained the term social participation or the keyword participation. We reviewed the matching articles and focused on those that reported development of instruments for medical, vocational, or mental health rehabilitation.

Domain description

Table 1 lists the initial 8 domains, the number of items we pooled and binned from the literature review and the qualitative review, the count of the winnowed items, and the final set of items that underwent cognitive testing. After binning and winnowing items, we revised the conceptual framework. From an initial set of 2273 items across domains, we winnowed this set to 291 items for cognitive testing. Table 2 provides examples of item content. We phrased each item as a simple, declarative statement that is either positively or negatively stated. Given the several ways in which we characterized environmental features, we used 5 types of rating scales as shown in table 3, including 2 variations of a frequency rating scale, one indicating capability using a dichotomous environmental attribution response option and the other consisting of a 5-point magnitude rating scale.

Table 2.

Examples of item revisions

Item Stem	Domain	Reason for Revision	New Item
In case of a health emergency, I can get the  information I need easily.	Access to information and technology	Emergency is too broad of a concept	Broken into 2 items:  In case of a health emergency, I can get the   information I need easily.  In case of a natural disaster, I can get the   information I need easily.
Websites with information are available in a  format I can understand if I need them.	Access to information and technology	Excessive wordiness	Websites are available in a format I can  understand if I need them.
Do you use any communication devices, such as a  voice synthesizer or communication board?	AT	Infrequently used device category	Do you use any equipment or devices to help you  communicate and interact with people; to see,  hear, or remember things; or to control your  environment?
I can use my <device> in a variety of settings.	AT	Confusion with settings	I can use my <device> in a variety of places.
How much difficulty do you have feeling safe  outdoors in your community?	Built and natural environment	Suggested rephrase to specify crime	How much difficulty do you have feeling safe due  to crime in your community?
How much difficulty do you have feeling safe in  buildings in your community?	Built and natural environment	Suggested rephrase, edit: buildings and add  emergency to be more specific	How much difficulty do you have feeling safe in  stores during an emergency?
My community offers self-help and social support  groups.	Services, systems, and policies	Suggested rephrase	My community offers support groups I can use.
Help modifying my home to make it accessible is  available if I need it.	Services, systems, and policies	Suggested rephrase	Help to make my home accessible is available if I  need it.
The people in my life are sensitive to the  challenges I face because of my disability.	Social environment	Confusion over the word sensitive	The people in my life understand the challenges I  face because of my disability.
Health care providers understand the needs of  people with disabilities.	Social environment	Ambiguity about health care providers	Health care professionals understand the needs  of people with disabilities.
How do you believe your financial situation is  today compared with other people your age?	Economic quality of life	Difficult comparison; idiosyncratic rating scale  (worse vs same vs better)	Item eventually dropped.
I have access to emergency funds.	Economic quality of life	Confusing	I have access to extra money in case of an  emergency.

Table 3.

Item response options

Category	Domain	Response Options
Frequency	Access to information and  technology	Never
	Social environment Systems, services, policies	Rarely Sometimes Usually Always
Frequency	Economic quality of life	Never Rarely Sometimes Often Always
Capability  (difficulty)	Built and natural  environment Yes—No	None A little Somewhat A lot Complete difficulty
Environmental  attribution Magnitude	Built and natural  environment AT	Yes No Not at all A little bit Somewhat Quite a bit Very much

Domain 1: Access to information and technology

Domain 1 pertains to knowledge and information technology. Information technology includes conventional devices and technology to transmit and receive information (eg, mobile or landline phones, computers, e-mail and Internet services). Usability of information includes factors that influence the ability to access and understand information and includes considerations of reading level, literacy, usability, transparency, and information finding. Although this domain does not include specialized communication or devices, it does include built-in, commercially available access features (eg, Microsoft control panel accessibility features).

Domain 2: AT

Domain 2 includes the use and availability of specialized devices to facilitate participation in daily activities (eg, mobility, personal care, work, leisure, home participation). These devices are typically specialized and not part of standard built environments. AT belongs to an individual and may be portable. Our definition of AT does not include features of the built environment in public spaces, universal design features (eg, ramps on public buildings), and Braille signage. People use AT to assist their performance across a variety of functional tasks. We developed parallel items for mobility, personal care, vision and hearing, and communication AT. Participants answer a filter question, “Do you use any mobility devices, such as a cane, walker, scooter, or wheelchair?” If the answer is affirmative, they name the device most often used and report on the availability and reliability of the device and its effect on participation. Participants who report that they need a device, but do not currently have it, answer a follow-up question on barriers to accessing AT. Participants who report that they do not need a device skip to the next section.

Domain 3: Built and natural environment

Domain 3 pertains to the natural and built factors in the environment, including architectural features of buildings, land development (eg, sidewalks and roadways), environmental features (eg, noise, crowds, indoor air quality), and cognitive and sensory cues. Built factors include aspects of neighborhood and community livability, such as safety, lighting, and access to community resources (eg, stores, banks). Natural features of the environment include climate and weather conditions, preparedness for natural disasters, and the means for addressing weather, climate, and emergency conditions. This domain includes geographic and topographic features of the environment (eg, outdoor air quality). We wrote items in pairs; the first item assesses how much difficulty a participant has in performing an activity, and the second item asks respondent if they attribute barriers to the environment (eg, precipitation, noise). For example, the first item of 1 pair asks “How much difficulty do you have feeling safe due to crime in your community?” The second item asks respondents to indicate yes or no on whether “The difficulty I have feeling safe is due to a high crime rate, inadequate number of police, or no emergency call boxes.”

Domain 4: Systems, services, and policies

Domain 4 pertains to social services, employment, education, housing, independent living, health care policies, and systems of delivery. It includes services provided by nonprofit, voluntary, and community agencies. Two subdomains emerged pertaining to economics and transportation. The economic subdomain assesses access to coverage of work and educational systems and resources and access to alternative disability and income subsidy programs, workers’ compensation, unemployment, and others systems (eg, food stamps, child care, affordable housing). The transportation subdomain pertains to public and private travel, including paratransit and adapted transit systems, and access, availability, affordability, and quality of these services. The items are introduced with the statement: “Many systems and services can affect participation at home and in the community. For each category below, tell us how these services and systems affect your participation.”

Domain 5: Social environment

Domain 5 includes content from the ICF support and relations and attitudes domains. Following the ICF model, we sought to measure both social support and attitudes. To measure social support, we adopted as legacy instruments the PROMIS social support item pools, which assess the availability of companionship, informational, instrumental, and emotional support. The PROMIS social support item pools have been tested in large samples of English- and Spanish-speaking adults and are intended for use across chronic illness, disability, and general population samples. For the current project we created an attitudes item set specific to disability by selecting and, when necessary, modifying items from existing instruments (eg, The Facilitators and Barriers Survey of Environmental Influences on Participations among People with Mobility Impairments and Limitations section 6-community environment: services and attitudes, Stigma Scale for Chronic Illness²⁵) and writing new items covering content identified by our qualitative data and input from content area experts. In keeping with the ICF, we included both positive and negative attitudes about disability. Therefore, item content includes marginalization, stigmatization, oppression, and discrimination and inclusion, acceptance, respect, and fairness. Also, similar to the ICF, items cover the attitudes of both individuals (eg, health care providers) and society as a whole. We excluded the attitudes of respondents themselves, reflecting self-stigma, because it did not constitute an environmental factor. The fourth article of this special issue describes the psychometric properties of this item pool in greater detail.²⁶

Domain 6: Economic quality of life

Domain 6 is based on the empirical results of consistent feedback from focus group participants regarding the importance of economic assets and the effects of these factors on participation. The ICF taxonomy code for financial aspects (e1650) is related to this construct. This domain is critical to individuals’ quality of life and serving as a barrier to participation; it reflects a distinctive construct that few environmental instruments measure.^27,28 This domain focuses on how financial resources influence satisfaction with one’s living situation, adequate and affordable health services, adequate and affordable food, affordable community recreational activities, and family and friend financial assistance. The fifth article of this special issue describes the psychometric properties of this item pool in greater detail.²⁹

Cognitive interviews

We recruited 15 people with stroke (n=4), TBI (n=5), or SCI (n=6) to complete cognitive interviews by telephone. The mean age was 46 years; 33% of the sample were women. Participants included blacks (54%), whites (33%), and Hispanics (13%). Although 80% had completed some postsecondary education, only 13% were employed; others were retired (40%), seeking employment (20%), or unemployed and not seeking employment (27%). Most were ambulatory (53%), although 47% used a walking device, and 47% used a wheelchair. Participants reviewed 50 of the 291 items; between 4 and 12 participants examined each item. For each newly developed item, we asked participants follow-up questions to ascertain comprehension, opinions about the suitability of the items, and revisions. We used this feedback to delete 17 items and revise 57 others. Revisions included narrowing and specifying the focus of concepts that were too broad, reducing the length of items, clarifying the meaning of ambiguous items, and reducing potential ceiling or floor effects. Table 2 provides examples of the revisions.

Discussion

We began this project by reviewing the social sciences, disability, and rehabilitation literature and identifying a large pool of items from a diverse literature describing environmental factors for medical, vocational, developmental, and psychiatric rehabilitation applications. It quickly became evident that this topic has generated considerable measurement interest across varied human service arenas, but existing instruments (eg, FABS/M) were narrow in scope, whereas others pertain to nearly everyone. None of the instruments that we reviewed used contemporary psychometric methods of instrument development; all relied on classical test theory methods or reported descriptive, item-level information. Therefore, we proceeded to develop a new measurement system that would describe and assess environmental factors. As such, an immediate challenge was to reduce the multiplicity of approaches and items to a conceptually concise and clinically practical set of items. We began with the ICF’s taxonomy of environmental factors and distinguished subcomponents of some ICF domains. In the case of the built and natural environment, we realized that these are inseparable aspects of the physical world. Consequently, 5 of the 6 domains correspond closely to the ICF taxonomy of environmental factors; the sixth domain, economic quality of life, reflects an important construct that reflects the resources individuals have that affect their participation.

Cognitive interviewing provided us with feedback that allowed us to reduce the 291 draft items to 274. The number of items per domain ranges from 14 (AT) to 82 (social attitudes). The AT items are used as a set to evaluate the use of mobility devices, equipment to assist with activities of daily living, and devices to interact with people and environments. Depending on an individual’s use of technology, a respondent would likely complete no more than 45 of the total 274 items. Feedback from persons living with the consequences of disability was critically important to reduce ambiguity and wordiness and enhance clarity and improve response distributions. The next step in transforming these item sets into functional item pools is to administer the items to a diverse group of individuals living with stroke, SCI, and TBI. We will use methods developed by the PROMIS project to explore the dimensionality of each domain using confirmatory factor analysis and item fit to models based on item response theory.³⁰ We envision developing computer adaptive tests and short-form versions to measure each domain to evaluate the influence of environmental factors on participation.

Study limitations

Results from this study are limited by the number of participants in the focus groups and the number and diversity of persons who participated in the cognitive interviews. Although we searched for literature in various aspects of rehabilitation (medical, vocational, intellectual, psychiatric), we may have overlooked some sources. Because of time constraints, we relied on focus groups that we conducted in earlier projects; these groups addressed multiple aspects of participation and environmental factors. The geographic sampling was limited to Colorado, Illinois, and New Jersey and drew participants primarily from urban and suburban communities. The consequences of underrepresentation of persons from rural settings are unknown. The persons completing cognitive interviews were from metropolitan Chicago; the extent to which an urban, Midwestern city limits perspectives on environmental barriers and facilitators is unknown. Future studies should recruit participants from a broader sample of communities.

Conclusions

A fully developed set of environmental factor items will support a variety of research and clinical applications. Clinicians are interested in identifying aspects of the environment that are potentially modifiable; their interventions could focus on problematic environmental factors that limit participation and life satisfaction. Clinicians and researchers are interested in acquiring a fuller knowledge of how environmental factors affect participation and quality of life. Disability advocates could use more specific and detailed information about environmental factors to lobby for elimination of barriers. The domain definitions and items that operationalize the domains will allow us to evaluate these environmental factor influences in a variety of applications.

List of abbreviations

AT

assistive technology

ICF

International Classification of Functioning, Disability and Health

PROMIS

Patient-Reported Outcomes Measurement Information System

SCI

spinal cord injury

TBI

traumatic brain injury