Evaluating the Generalizability and Reproducibility of Scientific Research

Over the past several months, the editorial team at the Red Journal has evaluated various initiatives aimed at improving the generalizability and reproducibility of our published research findings. To that end, we are instituting 2 enhancements to our submission and publication process. These include the introduction of an optional generalizability table for all articles and the inclusion of study protocols to accompany the publication of primary endpoints of clinical trials.

First, to the issue of generalizability. Just as we face disproportionate representations of populations across many sectors of our society, representativeness in scientific and medical studies is not immune. Events over the past 2 years have heightened awareness of the imperative to address societal disparities as a major public health concern.¹ In response, a number of scientific journals have published statements addressing aspects of equity, diversity, and inclusion in research and publishing.^2,3 Leading journals have issued forms requesting disclosures of representativeness in research findings,⁴ composition of research teams,⁵ or citationality.^6,7

A critical principle in translating research to practice is the generalizability of a study cohort to other populations and settings. However, limitations on generalizability are frequently not obvious or not reported.⁸ Subtleties in a given study’s applicability to nonrepresented demographics or geographies are often left undefined, with potential for detrimental consequences directly in the clinic or as continued exclusion from future research development. As an example, the emerging literature in artificial intelligence and machine learning demonstrates the propagated biases of samples including limited diversity.^9,10 In the realm of biology, the sex of cells is known to have a profound influence on experiments in certain tissues but is frequently unreported.¹¹ Likewise, genomic studies have drawn from databanks in which the vast majority of the samples have been collected from Caucasian or Asian patients.¹² Incidence, outcomes, and mortality related to various diseases differ based on socioeconomic or racial and ethnic differences,¹³ but evaluations of safety and efficacy have not often encompassed diverse populations in clinical trials.¹⁴

It is important to recognize that limitations in generalizability or reproducibility may result from mechanisms that operate differently across populations or from structural limitations in delivering therapies appropriately. For example, an intervention might be less effective in older adults due to a higher competing mortality that offsets any benefit. On the other hand, an intervention could be equally efficacious in older adults but less able to be delivered due to limitations related to expense, transportation, support, or other structural barriers. Although these issues lie outside the scope of our proposed table, thoughtful consideration should be given by investigators to the underlying reasons or rationale for inclusion or exclusion from a study or therapy, as interventions for these 2 scenarios would differ.

After extensive discussion among our section leaders and executive editors, we have developed a supplemental table allowing authors the opportunity to comment on the generalizability and reproducibility of their study findings to broader populations (see template and examples in Tables 1–4). The table, adapted from an example provided by the New England Journal of Medicine,⁴ has purposely been left open-ended to fit a wide range of study topics and approaches. Inclusion of the table will not affect manuscript review or acceptance and is considered a voluntary part of the submission process. It will be published as supplemental, downloadable material along with the online publication of an accepted article.

Table 1.

Supplementary table on generalizability of study population(s)

Condition	Description
Disease, problem, or condition under investigation
Relevant considerations of disease, problem, or condition in relation to:	Note any relevant considerations in boxes below:
Sex or gender
Age
Race or ethnic group
Geography
Other considerations
Study	Description
Overall assessment of generalizability of the study population

Table 4.

Supplementary table on generalizability of study population(s)—Clinical example

Condition	Description
Disease, problem, or condition under investigation	HPV-associated oropharyngeal squamous cell carcinoma
Relevant considerations of disease, problem, or condition in relation to:	Note any relevant considerations in boxes below:  From CDC statistics in the United States, 2013–2017 except as noted
Sex or gender	Whereas 16,200 (82%) cases are diagnosed in men each year, 3500 (18%) are diagnosed in women. The male-female ratio is similar across White, Black, American Indian and Alaska Native, and Asian and Pacific Islander demographic groups, and among Hispanic vs non-Hispanic demographic groups.
Age	The median age at diagnosis is 63 years among women and 61 among men.
Race or ethnic group	The numbers of cases (rates per 100,000 persons) in the United States are as follows: White, 17,635 (5.5%); Black, 1608 (3.6%); American Indian and Alaska Native, 119 (3.0%); Asian and Pacific Islander, 259 (1.3%); and Hispanic 1000 (2.5%) vs non-Hispanic 18,775 (5.4%).
Geography	The relative proportions of cases as classified by the 2012 Human Development Index (HDI) are: low- and medium-HDI countries: 28% versus high- and very high-HDI countries: 72%. (de Martel C, Plummer M, Vignat J, et al. Worldwide burden of cancer attributable to HPV by site, country and HPV type. Int J Cancer. 2017;141:664–670)
Other considerations	Among oropharyngeal cancers, approximately 50% to 60% of White patients are p16+ or HPV+ whereas 20% to 25% are p16+ or HPV+ in Black and Asian patients. Although Black oropharyngeal cancer patients have lower survival rates than similarly staged White patients, the difference is not present after adjustment for p16/HPV and smoking. (Ragin C, Liu JC, Jones G, et al. Prevalence of HPV infection in racial-ethnic subgroups of head and neck cancer patients. Carcinogenesis. 2017;38:218–229)
Study	Description
Overall assessment of generalizability of the study population	In this study of HPV-associated oropharynx cancer, the median age was 59 years, and 81% of patients were male and 91% were White, similar to the US population. There were low proportions of older adults and racial and ethnic populations other than White. It is unknown if outcomes would differ in these populations, although there is no established mechanism known to date. In this study, outcomes among women were slightly superior, but the mechanism for this is not known. There could be differences in the relative prevalence of different HPV strains or importantly tobacco/smoking exposure in other regions of the world that would affect outcomes.

We highly encourage authors to include this table with their new submissions. In pilot testing, our executive and section editors commented that completing and reading these tables added new insights about the backgrounds of the diseases being studied and specific perspectives that the articles bring forth to the literature. They felt it enhanced their understanding of the research and its potential pathways for further development. These insights do not in any way detract from an article but add to a greater appreciation of the research’s applicability and future possibilities. It should be noted that the goal is greater inclusivity and applicability across populations. The intention is not to imply that therapies should be restricted only to majority populations that have been explicitly studied or that differences in safety or efficacy should be assumed if there is no clear mechanism supporting this assertion. For example, there may be a low percentage of racial or ethnic minority persons in a study but no plausible mechanism for why they would not have a similar response to the treatment. In these cases, one would err toward greater inclusivity in giving a beneficial treatment in the absence of a reason justifying exclusion.

We recognize that this table is a small step in addressing the complex issues of equity, diversity, and inclusion in scientific research and publishing. The Red Journal has a history of striving for diversity and inclusiveness on our editorial board, seeking out a variety of voices and experiences that have made our journal and community stronger.¹⁵ We now hope to stimulate a conversation that will extend to a larger-scale assessment of how scientific research can be more thoughtfully designed to meet the needs of evolving populations, as well as the value that including diverse and informed perspectives on research teams can bring in expanding scientific research’s impact and relevance to all humans. Although we realize this table may be viewed as an extra burden or obligation, we urge our authors to use the opportunity to reflect on their research programs and brainstorm enhancements or expansions for the future.

Now, onward to clinical trial protocols. As a related editorial matter, we have received requests from reviewers and readers that the Red Journal make study protocols available when reporting clinical trials. This is considered a higher level of research reporting that is underused¹⁶ except at a few top oncology journals.^17–19 Controversies over publication of protocols include compromise of the trialists’ or sponsors’ proprietary information or confidentiality, outdated or incorrect versioning, or potential negative consequences for the research itself.²⁰ However, understanding on-the-ground conditions of study implementation and conduct is important in evaluating the reliability of scientific findings and their reproducibility in other settings.²¹ There is also value in education and transparency for the readership and public that is not to be discounted.

We began pilot-testing the optional submission of study protocols several months ago, finding enthusiastic uptake, and we have now transitioned to implementing this on a larger scale. For randomized phase 2 and phase 3 trials, submission of the study protocol as an accompanying supplementary file will be required. The article submission and protocol may be anonymized if the authors so choose, but they do not have to be. To be clear, trials of this specific type are permitted to be exempt from any anonymization given that they are almost always easily recognizable to our reviewers and editors. The protocol may be redacted to remove confidential or proprietary information before submission.

For other types of studies, such as nonrandomized, phase 1 or phase 4, or registry or quality-improvement studies, we highly encourage submission of a protocol supplement, although it is not required and if submitted will not be released to reviewers to preserve the conditions of double-anonymization. In these cases, supervising editors will carry out basic checks that the protocol is authentic and will release relevant sections to handling editors and reviewers upon request, if questions arise during the review process. If the article is accepted, the protocol will be published with the article.

After acceptance and upon publication of the parent article, all submitted protocols will be published as supplementary appendices available for viewing at the Red Journal website. If there are specific concerns about submission or publication of a protocol, authors may apply to the Editorin-Chief for an exemption.

The executive editorial team believes these 2 initiatives will inspire a higher quality of research and education at the Red Journal. We thank our numerous editors who overwhelmingly voiced their support and contributed to this collective development.

Table 2.

Supplementary table on generalizability of study population(s)—Physics example

Condition	Description
Disease, problem, or condition under investigation	Early-stage non-small cell lung cancer (NSCLC)
Relevant considerations of disease, problem, or condition in relation to:	Note any relevant considerations in boxes below:  Data below are based on NCI SEER (1975–2018)
Sex or gender	The incidence and mortality rates of lung and bronchus cancers in the United States are 12.2% and 14.9% higher, respectively, in males compared with females.
Age	The median age at diagnosis of lung and bronchus cancers in the United States is ~ 71 y.
Race or ethnic group	Both incidence and mortality rates of lung and bronchus cancers in the United States are approximately the same for Blacks and Whites.
Geography	For lung cancer, country-specific Human Development Index is correlated with age-standardized incidence and mortality, as well as GDP. For men, 22 and 30 countries showed declining incidence and mortality trends, respectively. For women, 19 and 16 countries showed increasing incidence and mortality trends, respectively. For men, average annual percent changes ranged from −2.8 to −0.6 and −3.6 to −1.1 for incidence and mortality, respectively, in countries with a declining trend. For women, average annual percent changes ranged from 0.4–8.9 and 1–4.4 for incidence and mortality, respectively, in countries with an increasing trend. For women, Brazil, Spain, and Cyprus had the greatest incidence increase. (Wong MCS, Lao XQ, Ho KF, et al. Incidence and mortality of lung cancer: Global trends and association with socioeconomic status. Sci Rep. 2017;7:14300)
Other considerations
Study	Description
Overall assessment of generalizability of the study population	In this small patient cohort, the median age was 67.7 years, and 85% of patients were male. Data on race/ethnicity were not readily available. All patients were from the United States. The imaging techniques and experimental findings of this study cannot be confirmed as translatable or achieved in all populations, due to the low proportion of women and unknown racial/ethnic composition and a lack of patients over 73 years of age, although there is not clear evidence that results would differ in these populations. Results of these techniques could be affected by different levels of tobacco/smoking exposure, which may produce variations in histology and disease presentations across US vs global regions.

Table 3.

Supplementary table on generalizability of study population(s)—Biology example

Condition	Description
Disease, problem, or condition under investigation	Pancreatic ductal adenocarcinoma (PDCA)
Relevant considerations of disease, problem, or condition in relation to:	Note any relevant considerations in boxes below:  Data below are based on NCI SEER (2014–2018) and WHO (2020)
Sex or gender	The incidence rate of PDCA in the U.S. is 15% to 60% higher in males compared with females.
Age	The median age at diagnosis of PDCA in the U.S. is ~70 years. Diagnosis before the age of 40 years is rare.
Race or ethnic group	Both incidence and mortality rates of PDCA in the United States are at least 10% to 20% higher in African Americans compared with other racial groups.
Geography	Pancreatic cancer is the seventh leading cause of cancer death globally, although incidence rates are rising worldwide. Lower reported incidence rates in African countries may be related to limited access to monitoring and oncology care.
Other considerations
Study	Description
Overall assessment of generalizability of the study population	This study was performed in 4 human pancreatic cancer cell lines, examined in cell culture and as tumor xenograft in a mouse model. Two cell lines (MIA PaCa-2 and Panc-1) were derived from a Caucasian male and 2 cell lines (KP-4 and SUIT-2) from an Asian male. Therefore, this study was not designed to provide information on cell lines from other demographic groups, such as females, and the potential influence of their genetic factors on radiation-induced biological outcomes in PDCA.