Exclusion of Non-English Speakers in Published Emergency Medicine Research - A Comparison of 2004 and 2014

Abstract

Background:

Non-English speakers (NES) as a proportion of the United States population have steadily increased in recent years. There remains substantial risk of excluding NES from research.

Objective:

To assess whether the percentage of emergency medicine (EM) studies that exclude Non-English speakers from participation has changed with time.

Methods:

In a structured fashion, the lead investigator analyzed all original research articles in Academic Emergency Medicine and Annals of Emergency Medicine retrospectively for 2004 and prospectively for 2014. An independent investigator conducted a blind review of a sample of articles to assess for interobserver agreement. Demographic data were analyzed using descriptive statistics. Chi-square, t-tests, and linear regression models were utilized; alpha set at 0.05. Cohen’s kappa calculated to assess interrater reliability.

Results:

We included a total of 236 original research articles. Overall, 11% excluded NES from research (10% AEM, 12% Annals). Cohen’s kappa (nonweighted) was 0.73. Comparing all articles in 2004 vs. 2014, research excluded NES 6% vs. 16% of the time respectively (P=0.02). This was not statistically significant when comparing year to year for AEM (7.3% vs. 14.5%; P=0.12) and Annals (6.7% vs. 19%; P=0.06) separately. Factors affecting NES exclusion included type of study design (P<0.001), geographic area (P=0.009) and hospital type (P=0.035). Interestingly, 42% of articles failed to mention language as an exclusion or inclusion criteria.

Conclusion:

We found that the percentage of articles excluding NES from EM research increased between 2004 and 20014. Further, many investigators do not report whether NES are excluded/included in their studies.

1. INTRODUCTION

Non-English speakers (NES) as a proportion of the United States population have steadily increased in recent years (1). NES are defined as individuals who self-categorize as not proficient with spoken English. Recently, a Census Bureau report using 2011 survey data representative of the US population over 5 years old found that over 60.6 million people (20.7% of US population) speak a language other than English at home. Of those individuals 22.4% (4.6% of US population) self-categorize as able to speak English “not well” or “not at all”. In total, there are 381 languages other than English spoken in US homes with Spanish or Spanish Creole as the most prevalent. These comprise 62% of the non-English speaking population. However, among all Spanish speakers in the US, 25.9% rate their ability to speak English “not well” or “not at all.” (1)

The dynamics of English proficiency across decades is ever changing. Between 1980 and 2010, there was a 158% increase in those speaking another language in the home, with only a concomitant 38% increase in total population growth. Among Spanish speakers, that increase was 232% (2). It is apparent that NES in the population are increasing in both real and proportional terms.

Amidst this disproportionate increase of NES and in effort to properly address its negative sequelae, President Clinton signed Executive Order 13166 in year 2000 outlined requirements for “Improving Access to Services for Persons with Limited English Proficiency.” Compliance requires entities receiving federal aid to make programs accessible to NES individuals and prohibits discrimination on the basis of language proficiency (2).

More specifically related to biomedical research, studies have shown that NES continue to present specific ethical and legal challenges to researchers. Institutional Review Boards (IRBs) have varying policy directives concerning how to equitably treat NES research subjects. Federal regulations protecting such individuals in research are similarly lacking (3). For example, one study reviewing 134 medical center websites found a wide array of consent requirements and procedures, and urged for more federal direction to ensure increased NES representation in research (4-6).

This is especially true in the field of emergency medicine (EM) research where data on exclusion or inclusion of NES is summarily lacking. This is particularly worrisome given the emergency department’s (ED’s) status as the safety net for the uninsured, poor, or otherwise underrepresented members of society. There is real risk that EM research is failing to include an already sizeable and apparently growing segment of the population, one that is vulnerable in its basic ability to communicate illness. It is impossible to address this issue if it has not been quantified at the most basic level.

Taken more broadly amongst the general medical literature, NES exclusion reporting remains wanting. At best, data is years old, but remains sparse and decade gaps frequently exist (4, 5). To further characterize NES exclusion rates over time, we retrospectively (calendar year 2004) and prospectively (calendar year 2014) evaluated a sample of EM literature with the hypothesis that NES exclusion rates were significant and worsening with time.

2. METHODS

Study Design

This was structured review of the literature to determine the frequency of language exclusions for subject participation in research as reported in articles during two years (2004 vs. 2014) for two major EM journals.

Setting

Study investigators conducted the literature review at Christus Spohn Memorial Hospital. Our facility is the base location for an emergency residency program. The institution’s IRB provided an exempt status for this protocol.

Study Protocol

All published original research articles in Academic Emergency Medicine and Annals of Emergency Medicine retrospectively for 2004 and prospectively for 2014 were analyzed (the two US-based journals for Emergency Medicine with the highest impact factors). Research articles were excluded for investigations performed at an international site, all meta-analyses, all systematic reviews, and any research not involving human subjects. Articles were assessed by the lead investigator in their entirety including abstract and methods sections specifically for exclusion or inclusion of NES. If exclusion/inclusion was inferred, this was noted. Papers without any language exclusions specified were recorded. A separate research assistant blinded to results and using a database not accessible to the original data collector performed convenience sampling of all articles from random months for both journals in both years.

We collected additional data points to further characterize the research articles. Types of study included randomized trial, nonrandomized trial, prospective cohort, retrospective cohort, case control, cross sectional, case series, case report, and surveys. Purpose of study characterized whether the study assessed a medication intervention, procedural intervention, risk assessment, or other. A research article’s geographic area was identified and coded according to Census Bureau regions: Northeast, Midwest, South, West, multiple, international, or not specified. The hospital location was also identified as either inner-city, suburban, rural, multiple, not specified, or not applicable, while a hospital’s designation as academic or non-academic was also noted. We recorded ED census when described.

Statistical Analysis

Demographic data were analyzed using descriptive statistics. Linear regression models, Chi-square and t-tests further tested our hypothesis with alpha set at 0.05. No interim analysis was performed. Agreement was measured using kappa to assess NES exclusion/inclusion. We hypothesized that there was a high and worsening exclusion rate of NES from EM research between 2004 and 2014.

3. RESULTS

We included a total of 236 original research articles, 134 from AEM and 102 from Annals. The journal article characteristics are summarized in Table 1. In 2004, AEM published 65 articles that met inclusion criteria while Annals published 60 that met criteria. For 2014, this number for AEM and Annals was 69 and 42 respectively. Analyzing by year only, 125 articles met inclusion criteria for 2004. There were 111 articles for 2014. Overall, 11.0% (26/236) excluded NES from research. By journal and for both years combined, 10.4% (14/134) of AEM articles excluded NES from research while Annals articles excluded NES 11.8% (12/102) of the time. This compared to a total of 47.0% (111/236) that included NES which broke down by journal to 44.8% (60/134) for AEM and 50.0% (51/102) for Annals.

Table 1.

Summary Data Journal Article Characteristics

Primary and secondary outcome measures are shown in Table 2. When comparing all articles meeting inclusion criteria across 2004 and 2014, NES were excluded 6.4% (8/125) and 16.2% (18/111) of the time respectively (P=0.02, 95% CI 0.017 to 0.184). However, when stratified by journal, there was no statistical significance. AEM articles excluded NES 7.3% (4/65) of the time in 2004 compared to 14.5% (10/69) in 2014 (P=0.12, 95% CI -0.024 to 0.192). For Annals, NES exclusion for 2004 and 2014 was 6.7% (4/60) and 19.0% (8/42) respectively (P=0.06, 95% CI -0.006 to 0.272).

Table 2.

Outcome Measures

Interestingly, many articles failed to indicate if NES were included or excluded in their research. Across both journals combined for 2004 and 2014, 41.9% (99/236) had no indication of NES inclusion or exclusion. This correlated to 41.6% (52/125) of all articles in 2004 compared to 42.3% (47/111) in 2014 (P=0.90). In 2004, 47.7% (31/65) of AEM articles and 35.0% (21/60) of Annals articles made no indication. For 2014, 42.0% (29/69) of AEM articles and 42.9% (18/42) of Annals articles made no indication. There was no statistical significance when comparing AEM articles with no indication for 2004 compared to 2014 (P=0.51). The same was true comparing 2004 to 2014 Annals articles making no indication of NES inclusion or exclusion (P=0.42). For those articles that did mention NES exclusion or inclusion, 23.4% (26/111) excluded NES.

We also assessed whether particular article characteristics such as type of study, purpose of study, geographic area, hospital location, and hospital designation. These findings are summarized in Table 3. We found that type of study (P<0.001), geographic area (P=0.009), and hospital designation (P=0.035) were associated with language exclusion of NES while the purpose of the study (P=0.781) and hospital location (P=0.076) were not.

Table 3.

Article Characteristics and Association with NES Exclusion

4. DISCUSSION

Our review of the literature revealed that there is a paucity of reporting on exclusion of NES individuals in research in general. Investigations looking at rates of exclusion of NES subjects are years sometimes decades old. (4, 5) In the face of increasing NES homes that data is now dated and without reassessing NES exclusion or inclusion in research today it is impossible to know if the problem persists. This is especially true in the field of EM research where we were unable to find any articles addressing the NES exclusion rates. This is particularly worrisome given the ED’s status as the safety net for the uninsured, poor, or otherwise underrepresented members of society.

We found a sizeable and worsening percentage of articles excluding NES from EM research which was statistically significant. From 2004 to 2014 exclusion of NES from two EM journals increased from 6.4% to 16.2%. This increase in exclusion from EM research trended towards significance when analyzing each journal individually. Certain research characteristics including type of study, geographic area, and hospital designation all correlated to exclusion of NES.

Surprisingly, many research articles during the two study years made no mention of NES inclusion or exclusion and were impossible to infer whether they did. While the percentage of research articles not reporting NES exclusion did not change with any statistical significance, the actual value still loomed large. Fully 41.9% of articles made no mention of NES exclusion or inclusion. Uncertainty regarding how protocols incorporate language proficiency is a known issue. In researching exclusion of vulnerable groups from clinical research, one article reviewed over 750 research protocols from a tertiary care facility over two years, yet they did not assess language proficiency.

Multiple studies have demonstrated a lack of protection for NES individuals. One study published in 1996 reviewed publications in US academic journals from 1989 to 1991 and showed 50% of studies excluded NES mostly due to translation issues or lack of bilingual staff (4). More recent research from 2007 notes that many studies neglect to report exclusion or inclusion of NES. Eighty-four percent of published articles in the British Medical Journal failed to address language accommodations at all. Fifty percent of those who did excluded NES research subjects /5). Our data showed over 40% of published articles in AEM and Annals similarly failed to address language. Of those articles that did address NES, almost a quarter excluded them. EM researchers need to be vigilant in addressing language exclusions for their research. Otherwise, we are still unclear on how much research excludes NES.

Prior research has evaluated how to increase inclusion of Spanish speakers in mail-based survey research, but there is a lack of rigorous studies examining how to increase participation in hospital based research (7). In fact, one study has even questioned whether translation or interpreter services undermines study validity (8). Instead of worrying about affecting validity by including interpretation services, the larger question is how much a study lacks validity for excluding a large and growing demographic of the US population. Even so, another study showed that among minorities, Hispanic children were “overrepresented in potentially stigmatizing research” while being underrepresented in clinical trials and therapeutic research (9, 10).

There is real risk that EM research misses important clinical research conclusions by failing to include an already sizeable and growing segment of the population, one that is vulnerable in its basic ability to communicate illness. Considering our role in treating all individuals at any time, EM research should focus on studying the full spectrum of individuals who seek care regardless of language ability.

5. LIMITATIONS AND FUTURE QUESTIONS

Our study has several limitations that warrant further discussion. NES inclusion was occasionally implied for some articles and this was determined to be an exclusion by the investigator. This could introduce subjectivity and question reliability of the data. To reduce this risk, we had a second data abstractor for a random sampling of articles of their choosing in order to perform inter-rater reliability calculations. Our Cohen’s kappa was 0.734 which is considered substantial. On the other hand, some articles had no information at all regarding language and it could not be implied that NES were included. Regarding scoring an article as excluding NES, it had to be explicitly stated in the article. There is risk that this was simply missed in reviewing the research article.

It remains possible that the years analyzed are not representative of EM research in general. Future analysis of subsequent years would validate our findings. Furthermore, the years used are somewhat arbitrary and the original research published therein conducted over a range of years, and sometimes longitudinally over many years rather than all within one year. We did not assess at what point research data collection began but instead when it was published. Assessing NES exclusion or inclusion while accounting for when research began would perhaps be more instructive on current NES exclusion trends. Finally, further exploration of NES exclusion in other specialty literature and general medical literature would provide a more complete picture of this worrisome trend that exists in EM literature (11-13).

6. CONCLUSIONS

Few studies have assessed NES exclusion from the medical literature in general. For EM specific literature, we were unable to identify any prior data on NES exclusion rates. Our novel findings evaluating NES exclusion from EM research show high rates of exclusion (11% overall) that have significantly worsened from 2004 to 2014 (6.4% to 16.2%). Perhaps even more worrying, over 40% of articles fail to mention language as an inclusion or exclusion criteria. In regards to NES, this creates a body of literature that leaves open those very research questions the medical field is trying to answer. For a population that in growth outpaces many other demographic groups, this should be cause for changing reporting habits on NES inclusion or exclusion and ideally researchers should strive to be more inclusive of NES in their investigations.