Advancing Research on Medical Image Perception by Strengthening Multidisciplinary Collaboration

Abstract

Medical image interpretation is central to detecting, diagnosing, and staging cancer and many other disorders. At a time when medical imaging is being transformed by digital technologies and artificial intelligence, understanding the basic perceptual and cognitive processes underlying medical image interpretation is vital for increasing diagnosticians’ accuracy and performance, improving patient outcomes, and reducing diagnostician burnout. Medical image perception remains substantially understudied. In September 2019, the National Cancer Institute convened a multidisciplinary panel of radiologists and pathologists together with researchers working in medical image perception and adjacent fields of cognition and perception for the “Cognition and Medical Image Perception Think Tank.” The Think Tank’s key objectives were to identify critical unsolved problems related to visual perception in pathology and radiology from the perspective of diagnosticians, discuss how these clinically relevant questions could be addressed through cognitive and perception research, identify barriers and solutions for transdisciplinary collaborations, define ways to elevate the profile of cognition and perception research within the medical image community, determine the greatest needs to advance medical image perception, and outline future goals and strategies to evaluate progress. The Think Tank emphasized diagnosticians’ perspectives as the crucial starting point for medical image perception research, with diagnosticians describing their interpretation process and identifying perceptual and cognitive problems that arise. This article reports the deliberations of the Think Tank participants to address these objectives and highlight opportunities to expand research on medical image perception.

Disease detection and diagnosis rely on diagnosticians (ie, anyone tasked with interpreting a medical image, including, but not limited to, radiologists and pathologists) searching for abnormalities in medical images. Across all image acquisition (from light microscopy to magnetic resonance) and presentation methods (eg, via viewing glass slides through a microscope or radiographs at computer workstations), diagnosticians are tasked with combining their vast medical knowledge with contextual information about the patient to interpret a high volume of complex images quickly and accurately. This is a challenging endeavor. Diagnosticians face many well-known difficulties, from time constraints (1) and information overload (2) to repeated interruptions (3) and burnout (4). These difficulties affect health-care quality and patient safety and can inflict financial burdens on practices (5-7).

Even in ideal circumstances, human observers inevitably make mistakes. Errors are a well-documented reality of radiology (8-10) and pathology (11) practice. False positives can lead to unnecessary treatments, and false negatives may mean that disease goes untreated. Although some errors may be due to imaging limitations, diagnosticians sometimes fail to perceive visible abnormalities. These perceptual and cognitive errors (11-13) reflect inherent limitations in human memory, search, interpretation, and decision-making capabilities (14). Medical image perception research can improve clinical care, most notably display characterization for digital imaging guidelines (15), optimizing environments for diagnosticians (16,17), and evaluating the impact of computer-aided detection on observer performance for US Food and Drug Administration approval (18). Understanding the perceptual and cognitive mechanisms underlying diagnostician performance can therefore help to improve diagnostic accuracy (11-13). However, research on perceptual factors has received far less attention and resources than research on new technologies.

To advance research on medical image perception and cognition, the National Cancer Institute (NCI) convened the “Cognition and Medical Image Perception Think Tank” with the goal to advance research on medical image perception and cognition. The Think Tank brought together radiologists, pathologists, researchers working in medical image perception and adjacent fields of cognition and perception, and representatives from interested federal government agencies (including NCI, National Institute of Biomedical Imaging and Bioengineering, US Food and Drug Administration, and US Department of Homeland Security). Here, we report on the Think Tank’s deliberations and highlight opportunities to enhance research in medical image perception and cognition.

Note that we use the term “medical image” to cover a wide range of imaging techniques, from images of individual cells to whole-body medical images, from glass slides to 3-dimensional computer renderings. In addition to detection and diagnosis, medical imaging is also used to guide surgical interventions. The term would also cover, for example, a dermatologist visually examining a patient’s skin.

Identifying Research Gaps

The first challenge for Think Tank participants was to identify research gaps and critical, unsolved problems in pathology and radiology from the diagnosticians’ perspective. Information overload, integrating large amounts of information from various sources, and artificial intelligence (AI) were identified.

Information Overload in the Digital Era

In radiology, the transition from analog to digital produced qualitative changes in image interpretation. A radiologist scrutinizing a single chest radiograph for a lung nodule is typically trying to find a target with a very low signal-to-noise ratio from a single image. The same radiologist reviewing chest computed tomography is more often searching for a target with a fairly high signal-to-noise ratio in a scan comprising hundreds of image slices. In addition, radiologists’ decisions now incorporate much more information from nonimaging sources, such as electronic health records and genetic panels. The development of radiomics (19) will allow extraction of nonvisual information from image data, which will have to be integrated into the diagnosis.

Pathologists face similar challenges moving from glass slides to digital images, which changes how the pathologist interacts with the slide. Modern pathology diagnosis goes beyond assessing the presence or absence of diseased tissues and now includes prognostic and predictive elements as well as the integration of ancillary stains and additional modalities such as molecular profiles (20). Pathologists aggregate all these predictive covariates and biomarkers with patient demographics and history to make integrated diagnoses used by clinicians in determining patient management (21). The number of inputs the diagnostician must consider keeps growing, intensifying time pressures.

Integrating Information From Various Sources

Workflow

Diagnosticians move from case to case, organ to organ, and image modality to image modality while dealing with texts, email, pager messages, colleagues, trainees, telephone calls, and other interruptions. Moreover, the task of the diagnostician is to integrate information from diverse sources, whereas current decision-support tools (eg, clinical algorithms, AI-supported findings) are typically narrowly focused.

Fatigue and Workload

As diagnosticians’ workload has increased (22), physical and mental fatigue among diagnosticians are at unprecedented levels, contributing to both perceptual and interpretive errors. Multiple studies have reported a decrease in detection accuracy because of fatigue (23). For instance, if a tired radiologist scrolls too quickly through multislice images, the act of displaying all slices on screen can produce the false impression that the case has been adequately scrutinized. Errors can also result from lack of focus, distractions, satisfaction of search (when detection of 1 abnormality interferes with detecting others), and/or the failure to record what is perceived. Errors caused by physical and mental fatigue become more apparent at the end of the day (24).

Artificial Intelligence

AI tools are routinely proposed as the solution to many challenges facing diagnosticians. However, there is a mismatch between what diagnosticians need from AI and how AI is typically developed. In news reports, AI solutions are often pitched as “outperforming” the diagnostician, implying that AI systems are on the cusp of replacing the diagnostician (25) or at least serving as an equivalent reader (26). This is a flawed analysis. First, as with prior computer-aided detection systems (27), an AI algorithm that performs well in a laboratory setting may not perform as well once deployed in real-world clinical practice. Diagnosticians may come to over-rely on computer recommendations (28,29) or, conversely, learn to ignore them (30). Second, training biases (31) and “off label” use of AI for tasks it was not designed for (eg, using AI as a primary reader when it is intended as a second reader) may cause interpretation errors. Finally, human diagnosticians are more than collections of image-interpretation routines. They perform noninterpretative functions, such as interacting with patient-facing clinicians, dealing with patient responses, and managing quality control and other administrative functions, none of which AI can yet perform.

A better approach is to consider AI as a trained assistant, making the diagnostician’s work easier by automating aspects of the data reduction process and presenting information that can aid the decision process. This “trained assistant” role has already gained clinical utility in some areas, such as interpreting Pap smears (32), identifying focal lesions on screening mammograms (as in computer-aided detection) (33), and looking for errors in prescriptions (34). Such tools are intended to provide optimal presentation of findings, ensure complete and precise evaluation of the information, improve accuracy, and reduce cost. This model leaves the task of integration to the diagnostician.

An AI “Diagnostic Cockpit of the Future”

A metaphor for how perception and cognition research can guide the integration of AI tools is the “diagnostic cockpit of the future” (35). During World War II, thousands of B-17 bombers crashed on landing without any sign of mechanical failure. Psychologists Paul Fitts and Alphonse Chapanis noticed that the controls for the landing gear and the flaps were identical, so that pilots were often mistaking one for the other. By changing the shape of the controls, they were able to avert many crashes (36). Similar human factor input could create a diagnostic cockpit that reduces medical error. This cockpit would include a set of tools to “aggregate, organize, and simplify medical imaging results and patient-centric clinical data” (35, p. 579). The aggregation of multiparametric data is typically not a strength of human observers but is well suited to computer applications.

Questions and Opportunities for Medical Image Perception Research

What are the effects of information overload on medical decision making and decision quality? Are there technical solutions, decision-making aids, or strategies that could be used to more efficiently integrate complex information? Can such decision-making strategies capture the ability of the patient’s clinical history to provide a useful set of differential diagnoses?

More research is needed on documenting and mitigating the costs of task switching. For example, can system-level dashboards that integrate tools minimize the burden of switching? What interventions might be available to detect and alleviate the effects of fatigue on interpretive accuracy?

Pathology has been understudied, relative to radiology, presenting many research opportunities. Although in some cases pathologists make binary assessments (ie, cancer vs not cancer), in most cases their assessments require more nuanced grading or subtyping. Because treatment decisions frequently hinge on tumor classification, interobserver variability among pathologists is an important problem. How do these classification errors change as pathology transitions to using digital images?

We know even less about perceptual and cognitive factors in fields such as dermatology (37,38) and ophthalmology. Meanwhile, the COVID-19 pandemic has highlighted the importance of telemedicine (39), which may pose novel cognitive challenges.

Addressing Clinical Questions Through Perceptual Research

Is medical image perception research on nonexpert observers clinically relevant? The relevance argument assumes that diagnosticians and nondiagnosticians have the same visual, perceptual, and cognitive faculties. Thus, the perceptual and cognitive bottlenecks found in nonexpert observers will also apply to diagnosticians. Expertise changes the use of these faculties but does not create a new visual system.

Clinical relevance also requires that artificial tasks and stimuli used in these studies capture at least some key elements of their clinical counterparts. Perceptual studies often trade the specificity and messiness of clinical settings for experimental control. Complex stimuli are simplified, narrative reports replaced with binary responses, clinical history is absent, and the life-and-death stakes of the clinic are eliminated. Many diagnosticians are skeptical of such abstracted medical image interpretation. However, such methodological choices are required to make progress on some topics because parametric studies with experts are often impractical. Expert participants need to be reserved for relatively short, carefully crafted experiments.

These constraints motivate a “reverse translation” paradigm (see Figure 1). The process begins with a dialogue between diagnostician and researcher (and diagnosticians are often researchers!), where the diagnostician identifies a problem in the clinical setting (just as the diagnosticians at the Think Tank did in the section on identifying research gaps). The researcher then abstracts out a basic question about the underlying cognitive and perceptual capabilities required and develops laboratory versions of the problem to study the basic science (“use-inspired basic research”) (40,41). These studies may use observer models or nonexpert observers because diagnosticians’ time is severely limited (42). After multiple iterations, working out the basic science, the researcher can return to the clinical setting to test the resulting hypothesis with a plausible, well-focused experiment with clinician observers.

Figure 1.

The reverse translation paradigm. Diagnosticians and researchers work together to identify relevant problems in the clinical setting (1). Perception researchers abstract the problem to create a laboratory version of the problem (2) and conduct experiments to study the basic science involved (3). From these studies, researchers derive a hypothesis (4), which can be tested in the clinical setting (5).

We stress that this reverse translation strategy can succeed only if researchers study questions that diagnosticians want answered. We propose that closer collaboration between diagnosticians and basic researchers is critical to ensuring that perception research investigates problems that are clinically significant (see Challenges and Solutions for Transdisciplinary Research below).

Case Study: Prevalence

Research on the prevalence effect provides an illustrative example: 1) Low prevalence has been identified as a problem for diagnosticians (43). 2) Perception researchers create visual search experiments as an abstract approximation of medical image interpretation. For example, observers might search for semitransparent images of tools (targets) among nontool objects embedded in noisy backgrounds (44). 3) By using artificial search tasks and nonexpert observers, researchers were able to vary prevalence from 0.078% (45) to 98% (46), measure the development of prevalence expectations (47), and more. 4) Multiple studies converged on the hypothesis that observers become less likely to call out targets (becoming more “conservative” in signal detection terms) as prevalence decreases. Consequently, observers are more likely to miss targets in lower prevalence than in higher target prevalence scenarios (48). 5) Returning to the clinical setting to test the hypothesis, this criterion shift has been shown to occur in both mammography (49) and cytopathology (50). Prevalence effects have wide implications. However, these studies would be implausibly long for diagnostician observers. Even if one curated thousands of medical images for a study at 2% prevalence, using alternative study designs to reduce the burden to study participants (51), recruiting multiple radiologists to read all those images would be difficult. The basic science of the interventions must be established in the laboratory before replications can be attempted in clinical settings.

Challenges and Solutions for Transdisciplinary Research

Today, multiple barriers hinder collaborations between perception researchers and diagnosticians. Medical image perception is an inherently transdisciplinary enterprise, requiring collaborations between diagnosticians, medical physicists, computer scientists, biomedical informaticians, engineers, and psychologists. Historically, MDs such as William Tuddenham and Harold Kundel (52,53) championed applying principles from psychology and other disciplines to advance medical image research (54). These collaborations were easier to initiate in an era when more perceptual psychologists were embedded in radiology departments with diagnosticians and physicists. These psychophysical researchers could become intimately familiar with clinical issues and more easily find collaborators. Together, physicists and psychologists exposed diagnosticians to medical image perception research and attracted radiologists to the subfield. Unfortunately, coexistence of multidisciplinary researchers within medical centers is less common now. Nevertheless, perceptual and cognitive challenges in medicine still require collaborations between diagnosticians and perception researchers, including cognitive psychologists, vision and oculomotor scientists, cognitive neuroscientists, and so forth.

Transdisciplinary teams face many challenges (55). The Think Tank identified specific challenges for medical image perception collaborations and research and proposed several solutions to address these challenges. A substantial challenge for both collaboration and participation in reader studies is competition for diagnosticians’ limited time. Collaborations should be designed to respect the diagnostician’s schedule, with projects developed around common interests of both parties. Incentives for diagnosticians’ participation should include grant funding, institutional support (eg, career path recognition), and authorship on publications. Additional incentives should include providing continuing medical education (CME) credit and/or CME in risk management for participating in cognitive research on medical image perception and decision making. To facilitate CME credit, an educational module should be created for participants, with objectives such as education on fundamental perceptual or cognitive concepts relating to clinical practice (eg, human vision, attentional mechanisms, sources of error, cognitive heuristics). Such CME models would also require that researchers share their results with participants on completion of the study. Specialty organizations like the American College of Radiology and the College of American Pathologists could enable collaborations by incorporating perception research into their quality assurance programs. The colleges could build up a “library of errors,” images that tend to lead to incorrect responses, to be used both as a teaching tool and resource for perception researchers.

Fundamental knowledge of relevant concepts in cognition and perception should be incorporated into diagnosticians’ training to foster the value of collaborations. These concepts are included in the noninterpretive skills section of the American Board of Radiology Core and Certifying examination, although there is typically little formal instruction on cognition and perception during residency and fellowship. CME credit, discussed above, should be tied to this content. It would also be helpful to revive the practice of providing training in the basics of medical image perception for panelists on relevant National Institutes of Health study sections. Joint funding opportunities with NCI, National Institute of Biomedical Imaging and Bioengineering, and the Agency for Healthcare Research and Quality could support collaborative research by including protected time for diagnosticians committed to such research and clinical mentoring for basic research scientists.

Academic researchers, especially those not based in a medical setting, have limited access to diagnosticians, who tend to be extremely busy. As a solution, “pop-up” perception laboratories at diagnostician conferences have been successful in recent years (56). A perception laboratory at a medical conference allows researchers access to a larger population of experts at a time when those experts are not occupied with their clinical work. Sustaining and strengthening these efforts will require partnering with national and regional medical conferences (eg, College of American Pathologists, American College of Radiology, American Society for Clinical Pathology, American Society of Cytopathology, Radiological Society of North America [RSNA], Society of Nuclear Medicine and Molecular Imaging, Association for Medical Ultrasound). Industry could collaborate with perception researchers to create a model “reading room of the future” at conferences and invite diagnosticians to different workstations that promote research outcomes. The next frontier will be platforms that allow medical image perception research to occur online. The development of such platforms will benefit from targeted funding and industry collaborations.

Beyond the recruitment issue, university medical centers and NCI-designated cancer centers should be encouraged to make medical image perception research a priority and to financially sustain perception laboratories, perhaps through NIH infrastructure grants. Collaboration often develops serendipitously, and serendipity depends on proximity. For perception researchers not embedded within medical centers, opportunities to network or promote research through invited talks are limited. One solution would be to create a match-making resource to identify perception scientists, diagnosticians, conferences, private practices, and medical departments who are open to collaborative research.

In response to Think Tank deliberations, NCI created a networking and collaboration website for medical image perception research (https://ncihub.org/groups/medicalimageperception/overview). Members of the site can create a profile, listing their name, affiliation, email, and a bio detailing their area of expertise or interest and a proposed project or potential research contribution, such as participating in reader studies, inviting researchers to give talks at their institutions, providing image datasets, generating research ideas and providing feedback, or becoming investigators.

Collaboration is more likely if diagnosticians know about perception research. Although the primary purpose of a pop-up laboratory, such as the NCI Perception Lab at RSNA, is to recruit research participants, these endeavors can also expose diagnosticians to perceptual research and help spark conversations between researchers and diagnosticians. Clinical research projects could be encouraged, perhaps by grant supplements, to include perception research as a secondary aim. Perceptual researchers should be encouraged to include outreach to clinical audiences in their dissemination plans.

Accessing medical image datasets can be a challenge for perception researchers. Curating image datasets is often time-consuming for diagnosticians and requires financial resources to undertake and sustain. Sharing images across institutions is difficult because of privacy and confidentiality risks and liability issues. Even image sets from large government-funded trials can be challenging to access and may need additional annotation for perception research. One way to address these problems would be to develop standardized templates for research collaboration and sharing agreements. Funding agencies could promote the use of such templates, much as they now encourage PIs to make datasets publicly available.

Existing image repositories (eg, NCI’s The Cancer Imaging Archive or the Alzheimer’s Disease Neuroimaging Initiative) may reduce the difficulties associated with accessing clinical trial image sets, but they often lack needed clinical context, appropriate annotations, diversity of pathology, and an efficient interface for navigating and searching for relevant images. The NCI PRISM project (57) is creating an infrastructure to integrate radiology and pathology images, which will include imaging features assessed by humans and by algorithms along with clinical context. Obtaining human annotations can be a challenge because of the effort and coordination required to carry this out. To address these challenges, NCI’s Cancer Imaging Program is funding the hosting of annotated diagnostic imaging from phase II and III clinical trials conducted by NCI networks on The Cancer Imaging Archive. The Crowds Cure Cancer project recruits radiologists attending RSNA to identify lesions and lesion characteristics and to provide details regarding image quality. This professional crowdsourcing approach is a model for annotating other medical image repositories. Involving perception scientists in project design could help to ensure that variables important to perception studies are coded as well. As AI research faces similar issues around annotation, this might be a good area for collaboration between perception science, AI, and medical imaging (58).

Within the cognitive science research community, there has long been an interest in medical decision making. However, there has been only modest collaboration between basic researchers with expertise in areas such as human judgment and decision making and researchers with interests in medical image perception. Diagnosticians are often confronted with classic decision-making problems (eg, integrating abundant and various forms of information), which have been extensively researched but not in the context of medical imaging. Closer interaction with judgment and decision research communities could offer insight to further advance medical imaging research.

Symposiums and workshops should be held at cognitive science conferences to engage wider audiences (ie, Society for Mathematical Psychology, the Society of Medical Decision Making, Psychonomic Society, Vision Sciences Society). As a model, consider the outreach efforts conducted by the International Society for Optics and Photonics to educate researchers on the diagnostician’s perspective and challenges. This effort includes demonstrations of diagnosticians interpreting case s, giving researchers insight into how diagnosticians read images.

Strong connections already exist between perception, cognitive science, and computer science. Computational vision is a robust research field that substantially overlaps with perception research and computer science. As AI solutions to imaging problems proliferate, collaborations between clinicians, perceptual, decision, and computer scientists should be encouraged.

Elevating the Profile of Medical Image Perception and Cognition Research

Creating a network of invested stakeholders (eg, diagnosticians, academics, biomedical informaticians, government and regulatory agencies, medical centers, industry, and conferences) is critical to support an awareness program promoting education, collaboration, and research. Here we identify key essential elements for achieving this goal.

Diagnostician advocates are needed to lend visibility and credibility and to reinforce the value of perception research to the medical community. This is important not just to facilitate the research itself but also to encourage adoption of research-based solutions to diagnostician challenges. One model for this network is the Alliance for Digital Pathology, a regulatory science initiative that brings together diagnosticians, researchers, government, and industry to identify key aims to advance digital pathology and provide an infrastructure for collaborative projects (59).

Medical image perception research would benefit from a more robust presence in diagnostician-focused journals. Although there are specialty journals that routinely publish medical image perception research, publication in high-impact radiology and pathology journals is uneven. Some of this may be the result of a lack of editorial infrastructure (eg, the absence of associate editors with expertise in the area makes it harder to obtain reliable peer review of submissions). The basic perceptual and cognitive literature would be enriched by review and opinion pieces by diagnosticians describing the perceptual and cognitive questions that arise in clinical work. Similarly, it would be helpful to have reviews of the relevant perceptual and cognitive literature in widely read clinical journals.

Future Goals and Strategies to Evaluate Progress

The overall goal of advancing medical image perception research is to improve patient care by reducing diagnostic errors, improving machine interfaces, and mitigating diagnosticians’ challenges. Research should yield best practices that can be incorporated into every aspect of medical image perception, including training, device and algorithm development, and clinical practice. How will we know if this initiative is bearing fruit?

Outreach can be tracked via the number of funding announcements, number of new principal investigators recruited into the field, and grants with transdisciplinary teams including radiologists or pathologists. Match-making and new collaborations can be tracked, in part, via the website mentioned earlier. Medical image perception research outcomes can be tracked in industry modifications to workstations or AI, reductions in errors, reductions in the time diagnosticians spend per case, and improvement in diagnostician attrition. Ultimately, these outcomes should be modeled to measure the overall improvement in potential lives saved and/or quality of adjusted life years gained. The baseline trajectory for the field should begin with 2014, the year that NCI’s Vision Science Problems in Medical Imaging Workshop was held. We can then compare progress to a counterfactual trajectory, extrapolated from prior trends, and assess what is needed to rectify and/or further improve the trajectories (see Figure 2).

Figure 2.

Funding trends for medical image perception and cognition research at the National Institutes of Health, 2008-2020. Black lines denotes the linear trend based on 2008-2014 (ie, before the Vision Science Problems in Medical Imaging Workshop was held), extrapolated to 2020.

Summary of Think Tank Deliberations

NCI’s “Cognition and Medical Image Perception Thank Tank” brought together diagnosticians, perception researchers, and representatives from government agencies and the medical imaging community to deliberate and identify courses of action to advance medical image perception research. Attendees pinpointed key challenges facing diagnosticians that could be addressed by appropriately designed perception research. Challenges include information overload, data integration from multiple sources, and AI. Advancing medical image perception research requires active multidisciplinary participation from diagnosticians, perception researchers, and the medical imaging community. Together, these fields can improve patient care by addressing the many diverse challenges facing diagnosticians.

Funding

Funds from the National Cancer Institute were used to convene experts for the Think Tank that informed the manuscript’s scientific scope.

Notes

Role of the funder: National Cancer Institute (NCI) staff had roles in the design of the Think Tank, contributed to the writing of the manuscript and the decision to submit the manuscript for publication. Moreover, the manuscript was reviewed and approved for submission through the manuscript clearance processes of both the NCI Division of Cancer Control and Population Sciences and the National Center for Complementary and Integrative Health.

Acknowledgements:We would like to thank Paige Green for her leadership and support, and Elizabeth Krupinski for comments on the manuscript.

Disclosures: MT: None. GB: Has received payment from genae, a contract research organization engaged for work on a project sponsored by the Hammanatsu Corporation. AC: None. PC: None. TD: None. ME: None. AF: None. BDG: None. MG: Multiple patents over the past 30 years that were licensed as a bundle through the University of Chicago Technology Transfer offices. Includes royalties from Hologic, GE. Scientific advisor to Qlarity Imaging. SMH: None. TSH: None. YVJ: None. BK: None. SMC: None. SM: None. LN: None. JS: Patent title: “System and Method To Quantify Tumor-Infiltrating Lymphocytes (TILs) For Clinical Pathology.” FS: None. SES: On the Board of Review for the GE-AUR Radiology Research Fellowship Program and received travel reimbursements to attend 1 Board and Mentoring meeting per year. BS: None. LS: None. ES: None. MT: None. JST: National Science Foundation paid to Vanderbilt University. ALVD: None. AMV: 1. Grant support from Siemens Healthineers and from the UT MD Anderson Cancer Center-Siemens Alliance. 2. Grant support from the Institutional Research Grant Program of the University of Texas MD Anderson Cancer Center (UTMDACC). 3. Grant support from the UTMDACC Cancer Center Support Grant (CCSG) Radiation Oncology and Cancer Imaging Program. 4. Grant support from the UTMDACC Radiation Oncology Strategic Initiatives (ROSI) Program. DW: None. JMW: None.

Author contributions: MT: Conceptualization, Visualization, Writing—original draft, Writing—review and editing. GB: Writing—review and editing. AC: Writing—review and editing. PC: Writing—review and editing. TD: Writing—review and editing. ME: Writing—review and editing. AF: Writing—review and editing. BDG: Writing—review and editing. MG: Writing—review and editing. SMH: Writing—review and editing. TSH: Conceptualization, Funding acquisition, Project administration, Visualization, Writing—review and editing. YVJ: Writing—review and editing. BK: Writing—review and editing. SMC: Writing—review and editing. SM: Writing—review and editing. LN: Writing—review and editing. JS: Writing—review and editing. FS: Writing—review and editing. SES: Writing—review and editing. BS: Writing—review and editing. LS: Writing—review and editing. ES: Writing—review and editing. MT: Writing—review and editing. JST: Writing—review and editing. ALVD: Writing—review and editing. AMV: Writing—review and editing. DW: Writing—review and editing. JMW: Writing—review and editing.

Data Availability

Data and code used to generate Figure 2 are available via the Open Science Framework, at https://osf.io/cpzyk.