Conducting Accessible Research: Including People With Disabilities in Public Health, Epidemiological, and Outcomes Studies

Abstract

People with disabilities are largely absent from mainstream health research. Exclusion of people with disabilities may be explicit, attributable to poorly justified exclusion criteria, or implicit, attributable to inaccessible study documents, interventions, or research measures. Meanwhile, people with disabilities experience poorer health, greater incidence of chronic conditions, and higher health care expenditure than people without disabilities. We outline our approach to “accessible research design”—research accessible to and inclusive of people with disabilities. We describe a model that includes 3 tiers: universal design, accommodations, and modifications. Through our work on several large-scale research studies, we provide pragmatic examples of accessible research design. Making efforts to include people with disabilities in public health, epidemiological, and outcomes studies will enhance the interpretability of findings for a significant patient population.

In recent years, the inclusion of traditionally underrepresented groups in research has received increasing attention, including racial and ethnic minorities, women, elderly individuals, and children.^1,2 This focus has grown as the scientific community has raised concerns over the appropriateness of translating research conducted with narrow groups of participants to the general population. The inclusion of women and minority populations in research has been mandated by the National Institutes of Health,³ and has been the subject of several large-scale initiatives, such as EMPaCT (Enhancing Minority Participation in Clinical Trials) and Project IMPACT (Increase Minority Participation and Awareness of Clinical Trials).^2,4

Despite these trends toward increased inclusion of minority groups in research, people with disabilities (PWD) continue to be excluded from health research.^5,6 At the same time, PWD are one of the greatest potential beneficiaries of health care services. It is estimated that health care for adults with disabilities amounts to roughly $400 billion per year, and represents approximately one quarter of health care expenditures in the United States.⁷ Estimates of health care expenditures are even higher among adult Medicare and Medicaid recipients,⁷ and children with disabilities incur 4 times the health care expenses of their nondisabled peers.⁸ Not providing equal representation for PWD in health research seriously limits the application of research findings for a significant patient population. Through the use of accessible research design, PWD can fully participate in research opportunities and contribute to advances in health care.

When defined under a model of functional impairment, estimates indicate that 54 to 60 million Americans live with disabilities, making people with disabilities one of the largest minority groups in the country.^9,10 Rates of identified disability incrementally increase with age, ranging from 2.3% for children younger than 3 years, 12.2% of school-aged children, 21% of those aged 15 years and older, and nearly 50% of those aged 65 years and older, indicating that, across the lifespan, disability is a near-universal experience.¹¹ For adults, 12.6% reported 1 type of functional limitation, and 8.2% reported disability in 2 or more domains (e.g., communicative, physical, vision, hearing, cognitive, emotional).¹¹ For children, the most prevalent conditions are neurodevelopmental or mental health conditions, such as attention-deficit/hyperactivity disorder, speech disorder, autism, or learning disability, followed by physical conditions.¹²

INCLUDING PEOPLE WITH DISABILITIES IN HEALTH RESEARCH

Adults with disabilities live with a thinner margin of health than their nondisabled peers, and have a higher incidence of chronic conditions and health-related disparities.^10,13 Chronic conditions cause disability, and, conversely, individuals with pre-existing disability are at higher risk for developing secondary conditions.¹⁴ When sampling working-age adults with chronic conditions, Gulley et al. found that 25% also reported a disability.¹⁵ Conversely, 80% to 90% of those with disabilities reported at least 1 chronic condition. For children with chronic conditions, it appears that rates of disability are similar, or approximately 21% to 22%.¹⁶

The mechanism behind the higher risk of developing secondary conditions appears to not be solely related to the pre-existing disability, but rather to malleable risk factors, such as obesity, physical inactivity, and smoking.¹⁰ Adults with disabilities also report higher incidence of social risk, including poverty, unemployment, and limited education, as well as difficulty accessing preventive health care and health services. Data on children with disabilities indicate that familial social risk (e.g., low income, racial/ethnic minority, inadequate insurance, low parental education) influences access to health care services,^17–19 and children with disabilities demonstrate higher rates of obesity and physical inactivity than peers.^20,21

Equal and just representation in health research is necessary to improve outcomes for PWD. The influence of risk factors (e.g., chronic conditions, poverty, physical activity, access to preventive care), in interaction with a pre-existing disability, bears investigation in large-scale research studies. Disability must be seen as a demographic factor, like age, gender, race, and ethnicity, to accurately translate research findings and improve health outcomes for this patient population. For additional references supporting the claims presented in this article, see Appendix B, available as a supplement to the online version of this article at http://www.ajph.org.

IMPLICATIONS FOR EXTERNAL VALIDITY

Evidence suggests that people with disabilities are not included in large-scale research studies.^5,6,22,23 When researchers do not include PWD in health-related research, it can lead to serious concerns about the external validity of a study. In some instances, PWD are the people most seriously affected by a health condition; conversely, having a disability represents a risk factor for developing secondary health conditions. Therefore, not explicitly making efforts to include PWD potentially skews research findings toward more “healthy” patient populations.

One way that PWD and other groups are excluded from research studies is through overly rigid inclusion and exclusion criteria. Feldman et al.⁶ examined the exclusion rate for children with disabilities from mainstream developmental research published in high-impact journals. The authors found that 66.7% of articles mentioning disability explicitly excluded those children; the rate was nearly 90% when they included articles that did not mention disability explicitly, with the assumption that no mention meant exclusion. Furthermore, the expert raters determined that in 54% of the cases in which children with disabilities were excluded, they could have participated without accommodations. The raters determined that 63% of studies could have included children with at least 1 disability type, when provided with accommodations, without compromising the integrity of the research study.

Van Spall et al.²⁴ examined the incidence of poorly justified exclusion and inclusion criteria in published randomized controlled trials in high-impact journals between 1994 and 2006 by using a series sampling technique. Poorly justified criteria included any stated criteria not justified as affecting the following areas: ability to provide informed consent, intervention or placebo would likely be harmful, intervention would likely be ineffective, or the effect of the intervention would be difficult to determine. The majority of trials (84.1%) contained at least 1 poorly justified exclusion criterion and one quarter of all exclusions were poorly justified in 61.5% of the randomized controlled trials. Specifically, of the studies examined, 11% excluded because of physical disability or functional status, 7.8% excluded because of cognitive impairment, 81.3% excluded because of medical comorbidities, 10.6% excluded because of language or communication barriers, and 72% excluded because of age (< 16 years or > 65 years).

Van Spall et al. noted that

The advantages of stringent eligibility criteria are achieved at the risk of excluding patients who may be more likely to represent the population treated in clinical settings and who would better test an intervention’s effectiveness.^24(p137)

For example, a large Medicare study found that only 13% to 25% of patients discharged with heart failure met the enrollment criteria for 3 large randomized controlled trials that have largely influenced treatment of heart failure.²⁵ The exclusion of PWD because of overly strict or poorly justified criteria is not only an issue of social justice and equity but it also limits the generalizability of research findings in real-world settings.

The studies of Feldman et al.⁶ and Van Spall et al.²⁴ provide some data for explicit (and unjustified) exclusion of PWD, chronic conditions, and other groups. However, it does not capture PWD who are implicitly excluded because of inaccessible research design. Just as stairs are not accessible to people in wheelchairs, research design may not be accessible to PWD. When PWD cannot access recruitment materials, read or sign consent forms, complete standardized interventions, access examination tables, or complete research measures, they are implicitly excluded from participation.⁵

Making conscious efforts to include PWD in health research is imperative to provide equal and just representation for nearly 20% of the US population. High-quality scientific evidence that includes PWD is warranted to guide decisions related to risk, prevention, and treatment of health conditions. Through both explicit and implicit inclusion of PWD, we can enhance study design and conclusions.^5,23,26 The use of “accessible research design” provides a blueprint for the inclusion of PWD in mainstream health research.

PURPOSE

This article provides a conceptual basis for accessible research design, as well as pragmatic examples of large-scale health research incorporating accessible research design. Williams and Moore²³ have provided an excellent overview and recommendations for Universal Design of Research for people with disabilities. In this article, we seek to broaden the idea of accessible research by considering 3 levels of implementation: (1) universal design, (2) accommodations, and (3) modifications. The pragmatics of conducting accessible research are derived from the work of Williams and Moore, as well as our work on several large scale studies: Patient-Reported Outcomes Measurement Information System (PROMIS), the National Institutes of Health (NIH) Toolbox, the National Children’s Study, and the National Health and Nutrition Examination Survet (NHANES; see Appendix A, available as supplement to the online version of this article at http://www.ajph.org, to learn more about these studies).

The focus of this article is on mainstream quantitative epidemiological, public health, and outcomes research; a variety of resources are available for the reader interested in accessible research design for studying PWD with qualitative research^27,28 or surveys.²⁹

In addition, strategies outlined in this article focus on the nonbiological mechanisms related to research, including recruitment, consent, intervention, and measurement. Clinical or biomedical research might involve processes that are confounded by certain underlying causes of disability. For example, a new chemotherapy might interact with specific types of underlying conditions in ways that could have an impact on the outcomes of the study. Researchers should be conscientious about exclusion criteria to not unjustifiably exclude those with medical conditions unrelated to the biological process they are studying. The use of accessible research design will provide access for all eligible participants and ensure that those with disabilities have equal opportunity to participate in scientific research.

ACCESSIBLE RESEARCH DESIGN

The term “accessibility” is used to refer to a state in which an individual’s functional capacity and the functional demands of an environment are matched so the individual can effectively complete an activity. For example, an individual who uses a wheelchair can enter a building with a ramp because the demands of the environment (allows wheeled mobility) match the functional capacity of the individual (mobile with a wheelchair). However, when a ramp is not available, a person who does not walk cannot complete the activity because the environmental demands (climbing stairs) do not match the functional capacity of the individual.

In the context of research, lack of accessibility can affect a person’s participation in research activities in the areas of recruitment, consent, intervention, and measurement. The prospective design of the research study can be enhanced to include people with a wide range of functional capacities through the use of universal design, accommodations, and modifications. These 3 approaches are on a continuum from most broadly applicable to more narrowly focused. In other words, universal design allows for access to a broad range of people; accommodations may need to be provided to a smaller subset of people; and, finally, modifications are used when universal design and accommodations are not effective or appropriate.

Universal Design

Universal design means “designing all products, buildings and exterior spaces to be usable by all people to the greatest extent possible.”^30(p2) Universal design has been applied to a diverse array of environments and situations, including architecture, learning, Web-based interfaces, and playgrounds. Williams and Moore²³ proposed the term “Universal Design of Research,” which they defined as the “design of research so that all people can be included as potential participants, to the greatest extent possible, without the need for adaptation or specialized design.”^23(p3) Using principles of universal design provides varying ways for people to participate in the research, which increases usability for people both with and without disabilities.

Table 1 outlines pragmatic strategies throughout the research process that would be considered “universal design.” These strategies are by no means exhaustive, but rather are meant to raise awareness of strategies that can be used with people with a wide range of abilities. Many of these strategies address more than 1 type of functional concern, which is the benefit of universal design. The 7 categories of functional impairments listed in Table 1 are the most relevant to consider when one is assessing accessibility.

TABLE 1—

Strategies for Universal Design for Conducting Accessible Research

Strategies	Hearing	Vision	Color Discrimination	Speech	Hand Control	Reading Impairment	Mobility
Recruitment: presentation of information
Large, dark print with ample white background		√	√
Audio (in person or via video, Internet, or radio)		√				√
Internet sites compatible with screen readers	√	√				√	√
Recruitment: mode of response
Training staff in use of TTY and video relay services	√			√
Allow responses via Internet sites accessible to screen readers	√	√		√		√
Provide a telephone number for responses		√
Consent: presentation of information
Sending the consent form before initial meeting to review via accessible files	√			√		√
Provide consent forms written in plain language, including only essential words, and minimal in length						√
Consent: mode of response: allow electronic signature, videotaped verbal consent, or proxy consent		√			√
Accessibility of facility
Make sure buildings, rooms, equipment (tables, chairs, and examination tables) meet ADA standards		√			√	√	√
Provide height-adjustable tables to accommodate wheelchairs							√
Transportation
Provide location near accessible public transportation		√			√		√
Provide funding for accessible transportation		√			√		√
Measurement: presentation of information
Provide captions for audio	√
Make sure both visual and auditory information can “stand alone”	√	√	√			√
Provide large print on ample white background		√	√
Measurement: mode of response
Allow time for TTY or video relay services for deaf individuals when using telephone surveys	√
Multiple means of logging a response: using keypad arrows, mouse, track ball, or pad					√
Large targets on touch screen that do not require dragging		√			√
Consider tests that are not timed		√		√	√	√
Make sure colors are not the only way of conveying test information; reduce use of red, green, or blue; use colors with high contrast between each other.			√
Interventions: presentation of information
Large print on white background		√	√
Audio (in person or via video or Internet)		√				√
Internet sites and word or pdf files compatible with screen readers	√	√				√	√
Interventions: accessible technology: providing technology for interventions that provide multisensory and easy input (e.g., talking watches, pedometers with large displays, electronic diaries that allow for text or voice input)23	√	√	√	√	√	√	√

Note. ADA = Americans With Disabilities Act; TTY = teletypewriter.

Familiarity with the population of interest is key to knowing which accessibility features of universal design are most relevant. For example, as Moore²⁶ outlined, for patients with diabetes, there is a significant incidence of visual, hearing, and mobility impairments. Keeping these functional impairments in mind can help determine which aspects of the research process may be most inaccessible in the participant population. For studies with a diverse sample, such as NHANES, providing the broadest amount of universal design features helps capture data from the most diverse group of participants. Universal design examples from NHANES are included in Appendix A.

Accommodations

Even when universal design is applied throughout the research process, there may be times when accommodations are needed to enable equal participation for PWD. Accommodations change how the task is accomplished so that respondents are able to participate in a task that they would otherwise be unable to complete. In other words, accommodations remove confounding influences of the assessment format, administration, or ways of responding. Accommodations are designed to keep the construct or essential elements consistent, while eliminating difficulties associated with functional deficits.³¹ For example, a person with limited functional hand control may not be able to complete a pencil-and-paper memory test, yet the person’s memory may be intact. Allowing a proxy or some other means for logging a response would be an appropriate accommodation that will allow the participants to demonstrate their memory (the core construct) while removing the confounding influence of difficulty with hand control.

Accommodations are commonly used in educational assessment for children with disabilities, and have involved making changes to (1) the setting of a test, (2) the presentation of a test, (3) the mode of response to a test, and (4) the timing or scheduling of a test.³² Extrapolating to the health research setting, making changes for an individual with a disability can involve the physical setting; how the measures, information, or interventions are presented (e.g., visual, auditory); the mode of response (e.g., verbal, pointing, clicking with mouse, writing); or the schedule (e.g., allowing for shorter testing sessions, or more time to complete a test).

The box on page 2141 outlines some strategies that address the setting, presentation, mode of response, and timing or scheduling that can be used throughout the research process. Like the universal design table, these are not exhaustive, but rather are meant as examples of common, pragmatic accommodations. We have included accommodation examples from NHANES in Appendix A.

Strategies for Accommodations for Conducting Accessible Research

Accommodations
Setting
• Provide the option for a quiet room; provide headphones for listening to auditory information (hearing).
• Provide assistance for navigating busy or cluttered areas; clear obstacles to the extent possible (vision, mobility).
Mode of presentation
• Provide or allow sign language interpreter (hearing), Braille technology (vision), screen reader (vision).
• Read information to participant or provide read-aloud technology (vision, reading).
• Provide alternative materials in large print (vision).
Mode of response
• Allow written responses or verbal responses (hearing, vision, hand control).
• Allow proxy to log response on paper-and-pencil tests or electronic media, keeping in mind that this may not be a good option for sensitive information (vision, hand control).
• Allow use of a communication device or a familiar friend or relative to accompany participant and serve as “interpreter,” provided the topic is not of a sensitive nature (speech).
• Consult expert on assistive technology (AT) if standard computer input options do not work (e.g., touch screen, mouse, keypad), or allow participant to use own AT (hand control).
• Provide an adaptive intervention that allows alternative ways of engaging in interventions that still preserve essential elements and internal validity (e.g., engaging in upper-extremity exercise in a wheelchair instead of standing). For an overview of adaptive intervention, see Moore.26
Scheduling or timing
• Be aware that use of accommodations may require more time.
• Participant may fatigue easier, needing sessions divided over several days or frequent breaks.
• Offer scheduling later in day as self-care routine in morning may be extensive.
• Provide flexible scheduling around transportation needs.
• Create priorities in protocol if all testing cannot be completed.

Modifications

Ideally, if universal design, accommodations, or both are provided, people with disabilities should be able to participate in some or all research-related activities. Modifications represent a further alteration in standardized process, in that they may change the nature of the construct being assessed.³¹ Modifications change the way in which the measure or test is given, or provide an alternate measure. For example, a modification might be to administer a sensitive measure (e.g., depression or sexual practices) via interview instead of using the electronic version the majority of participants are using. When an intervention or assessment is modified, it may be essential to determine whether the essential elements, or constructs, remain the same. For example, using a proxy to log responses in a timed recall test may be considered a modification if it interferes with measurement of the central construct (speed of recall).

Modifications come with internal validity concerns. When one is using alternate tests, there is concern regarding measurement equivalence.³³ Modified outcome measures are increasingly being examined for different disability populations, such as the 36-Item Short-Form Health Survey³⁴ and the PROMIS Physical Function Scale³⁵ for wheelchair users. These modified measures can aid researchers in choosing comparable measures for their population of interest.

PRACTICAL STUDY ISSUES

Early consideration of the accessibility needs of diverse users helps ensure that study materials and protocols are able to accommodate as wide a range of participants as possible. Generating a protocol that includes variations for people with disabilities provides guidance for research personnel, and eliminates the need for on-the-spot decisions that may alter the internal validity of the study. Protocols can and should include information on screening for accessibility needs, accessible research design (including universal design options, accommodations, and modifications), training, and quality control (including documentation of alterations in protocol).

Screening

Screening research participants for accessibility needs as part of the standard intake protocol can help ensure that the study team is adequately prepared to provide appropriate accommodations and supports. Screening provides the time for the study team to brainstorm for and plan accommodations and make necessary provisions to the testing protocol (e.g., additional time, specialized equipment, availability of a second administrator).

Screening should begin with initial contact with a potential participant. Language should be built into recruitment materials and consent forms informing participants that accommodations are available for people with disabilities, and providing a contact person for those accommodations. People with disabilities are experts on their needs, and this more “informal” self-identification process can be extremely valuable in planning for accommodations and modifications.

Once a participant is contacted, a set of guiding questions should be developed to identify any needs related to the functional requirements of the study protocol (e.g., reading, using a touchscreen, extended testing time). By emphasizing the functional requirements, participants are able to identify needs without necessarily providing a diagnostic label; for example, a parent may indicate the need for breaks for a child with attention-deficit/hyperactivity disorder. The box on this page provides some sample questions based upon our work with the NIH Toolbox measures; however, these will vary according to the functional requirements of the study in question.

Sample Prescreening Questions for Conducting Accessible Research

Vision	Can you read typical printed material (e.g., books, pamphlets) with or without glasses?
Hearing	Do you have any difficulty hearing during typical daily conversation?
Fine motor	Do you have any difficulty using a touchscreen, such as an iPad or tablet?
Mobility	Do you use any assistive devices to get around, such as a walker, cane, or wheelchair?
Color discrimination	Have you ever been told you are colorblind?
Speech	Do you have any difficulty speaking so that others can understand you?
Reading	Do you have any difficulties with reading that we should be aware of?

Finally, a set of more formal screening measures may be built into the study protocol to identify those who may have difficulty with the functional requirements of the study procedures. For example, in the NIH Toolbox computer-based tests, participants are guided through some training screens before beginning the actual measure. Should a participant have difficulty with the training items—for example, because of decreased ability to use a standard mouse or inability to read or see text—then specific universal design options, accommodations, or modifications may be implemented. Other options for formal screening might include a basic vision screener, reading a simple written passage, or using a touch screen.

Documentation of Accessible Research Design

The study protocol should guide research administrators in the process of utilizing universal design options, accommodations, and modifications once participants have been identified with having a disability that affects their ability to participate in study procedures. For example, in NHANES, once persons are identified as having a mobility impairment that has an impact on their ability to stand for the height measure, the administrator is guided to move on to other anthropometry protocols. In the NIH Toolbox, an accommodations manual was developed that outlines general points for working with PWD, as well as functional demands of each measure, potential accessibility issues, and guidelines for accommodations.³⁶ A sample of accessibility guidelines for a cognitive test (Oral Reading Recognition) can be found in Appendix A.

Preplanning for disability accommodations can also help study teams identify a minimal data set based on high-impact results or key end points. In NHANES, study domain experts were asked to identify those elements that were critical for assessment in advance. Therefore, if unexpected complications arose, the study team had clear guidance on how to proceed.

Training, Quality Control, and Documentation

Although disability is a fundamental part of the human experience, many health care professionals are not well trained in how to effectively work with people with disabilities. A 2009 report by the National Council on Disability indicated that

The absence of professional training on disability competency issues for health care practitioners is one of the most significant barriers preventing people with disabilities from receiving appropriate and effective health care.^37(p13)

Consistent with the recommendations of Williams and Moore,²³ when one does not know how to include someone with a disability, one should consult someone who does (i.e., the potential research participant, another person with that disability, the child’s parent or guardian, or a health professional who works with persons with disabilities).

Just as adverse events are documented, so should alterations in protocol because of disability. Consistently documenting when alterations to the testing protocol are made is important for interpretation of results, as well as quality control. By documenting these alterations, researchers can better understand whether an alteration might have an impact on scoring or be related to an outlier in the data. Documentation of the testing alterations and accommodations can also ensure consistency on repeated assessments in longitudinal studies by ensuring the same accommodations are provided on subsequent visits. Key to this documentation is understanding whether alterations in protocol are considered “standard” or “nonstandard” administration. Universal design and accommodations do not present changes in the core task demands or the primary construct being assessed. As such, these alterations should be documented as “standard administration” with a note regarding the type of accommodation used. If an alteration in protocol has the potential to alter core task demands, it would be considered a modification and should be documented as “nonstandard administration.” Finally, if a participant is screened out of a measure, the reason should be documented. See Appendix A for examples of standard and nonstandard administration for the Toolbox Oral Reading Recognition test.

FUTURE DIRECTIONS

Future research should include systematic evaluation of the impact that reasonable accommodations and accessibility features have on derived scores in people with and without disabilities to evaluate the concept of differential boost. The concept of differential boost is based on the notion that test takers should not receive an undue advantage from reasonable accommodations.³⁸ To use an example from education, many individuals, regardless of their disability status, may benefit from extra time on an essay-based examination; therefore, providing a student with a disability with an accommodation of extra time may confer an unfair advantage. In contrast, provision of a screen reader is likely to only benefit students who are blind and really need these technologies. The notion of unfair advantage is a major barrier to the acceptance of accommodated versions of assessments by end users. Rigorous evaluation with disabled and nondisabled participants is recommended to evaluate equivalence, but is rarely done.

Research is also needed to develop and evaluate the equivalence of alternate measures for people for whom the traditional measure is inaccessible. For example, many older adults experience visual decline and blindness because of conditions such as glaucoma and macular degeneration. At the same time, older adults also have high rates of cognitive deficits, attributable to both age-related functional decline and pathological conditions such as Alzheimer’s disease. There is thus a critical need to develop equivalent nonvisual cognitive assessments to measure cognitive decline and dysfunction in this population.

To document the need for greater inclusion of people with disabilities in epidemiological, longitudinal, and clinical research, there is a need for the systematic evaluation of attrition rates in large cohort and clinical studies to determine the impact of inaccessibility and overly strict inclusion or exclusion criteria. As demonstrated in NHANES, there is also a need for systematic and evidence-based decisions to develop minimal data sets for measures. Initiatives such as the World Health Organization’s International Classification of Functioning, Disability and Health Core Set and the National Institute of Neurological Disorders and Stroke’s Common Data Elements Initiatives seek to identify the most critical data for including in clinical research with target populations.^39,40

Public health must begin to recognize the importance of the inclusion of PWD in mainstream health research studies. There is a critical need for PWD to be recognized as a disparities population; as such, disability should be treated in research as another demographic factor, such as ethnicity, sex, or age. Federal funding agencies, such as the NIH and National Institute on Disability, Independent Living, and Rehabilitation Research must emphasize the importance of inclusion of people with disabilities, not only as a civil rights issue but also to enhance the scientific integrity and interpretability of research findings to real-world clinical and policy applications. Through the use of justifiable inclusion and exclusion criteria, and accessible research design, we can begin to include PWD and start to address health disparities in the 54 to 60 million people living with disabilities in the United States.