What Constitutes Evidence? Colorectal Cancer Screening and the U.S. Preventive Services Task Force

Abstract

The United States Preventive Services Task Force is perhaps America’s best-known source of evidence-based medicine (EBM) recommendations. This paper reviews aspects of the history of one such recommendation—screening for colorectal cancer (CRC)—to explore how the Task Force evaluates the best available evidence to reach its conclusions.

Although the Task Force initially believed there was inadequate evidence to recommend CRC screening in the 1980s, it later changed its mind. Indeed, by 2002, it was recommending screening colonoscopy for those aged 50 and older, “extrapolating” from the existing evidence as there were no randomized controlled trials of the procedure. By 2016, due in part to the use of an emerging analytic modality known as modeling, the Task Force supported four additional CRC screening tests that lacked randomized data. Among the reasons the Task Force gave for these decisions was the desire to improve adherence for a strategy—screening healthy, asymptomatic individuals—that it believed saved lives.

During these same years, the Task Force diverged from other organizations by declining to advocate screening otherwise healthy Black patients earlier than age 50—despite the fact that such individuals had higher rates of CRC than the general population, higher mortality from the disease and earlier onset of the disease. In declining to extrapolate in this instance, the Task Force underscored the lack of reliable data that proved that the benefits of such testing would outweigh the harms.

The history of CRC screening reminds us that scientific evaluation relies not only on methodological sophistication but also on a combination of intellectual, cognitive and social processes. General internists—and their patients—should realize that EBM recommendations are often not definitive but rather thoughtful data-based advice.

The United States Preventive Services Task Force is perhaps America’s best-known source of evidence-based medicine (EBM). Since its founding in 1984, the Task Force has evaluated hundreds of screening interventions using what it believes to be the best available evidence. Its decisions have been termed “scientific, transparent, and untainted by conflict of interest and political influence”.¹ The Task Force has explicitly not included costs of screening in its deliberations. However, beginning in 1997, Medicare has employed Task Force conclusions to help decide what it would and would not fund.² Since 2010, the Task Force’s recommendations have also been used to determine which screening tests will be funded by the Affordable Care Act—giving the group even more importance.³

There is often a sense that EBM brings objectivity to medicine, but merely terming one’s evaluations “evidence-based,” and employing a hierarchical strategy that prioritizes data from randomized controlled trials (RCTs), does not make the process straightforward. For one thing, within the world of EBM there are active debates about the value of various scientific study designs, such as the newer concept of microsimulation modeling. In addition, there are often disagreements about what certain data allows one to conclude. Finally, Task Force members, although renowned scientists formally trained in EBM, nevertheless bring opinions and value judgments to discussions of screening interventions.

This paper uses colorectal cancer (CRC) screening as a case study of how the Task Force has applied its rules of evidence. It is not meant to be a comprehensive account of how the Task Force has addressed CRC screening. Indeed, over the past three decades, the Task Force has evaluated CRC screening six times, and there are hundreds of papers that address its recommendations and reasoning. Nor do the debates discussed here exactly mirror those surrounding other screening tests. Nevertheless, the basic theme addressed here—the complicated ways in which the Task Force determines which types of evidence it believes are adequate or inadequate—is highly relevant for all organizations generating EBM recommendations as well as health professionals and patients trying to understand them.

EVIDENCE-BASED MEDICINE

A key progenitor of EBM was the growing use after World War II of RCTs to evaluate clinical interventions. Prior to this, physicians often used case series, expert opinions and retrospective assessments that suffered from a series of biases and assumptions.⁴ The growing field of clinical epidemiology, which studied the determinants and effects of clinical decisions, provided an institutional base for researchers studying evidence.⁵ In 1971, physician and epidemiologist Archie Cochrane published Effectiveness and Efficiency, which argued that most medical interventions had no clear scientific justification and pleaded for use of RCTs.⁶ In the 1970s and 1980s, physician and bioengineer David Eddy wrote several influential articles introducing new analytical methods for evaluating screening and other tests.⁷ The term evidence–based medicine is usually traced to David Sackett and his colleagues at Ontario’s McMaster University in the early 1990s.^8–10

Both the Canadian Task Force on Preventative Health Care, established in 1976, and the U.S. Preventive Services Task Force, which emerged eight years later, built on this work. There was a hierarchy of evidence, with RCTs at the top and other more potentially biased studies further down. In later years, meta-analyses, which evaluate multiple studies of the same intervention, also became essential for the U.S. Task Force to best evaluate whether interventions worked. Sackett and other experts emphasized that applying EBM to patient care was not “cookbook medicine” but rather a blend of the “current best evidence,” “clinical expertise” and “patient values”.¹¹ An early Task Force paper concurred, adding expert opinion, costs, medicolegal issues and reimbursement as important factors it used when arriving at its decisions.¹²

Importantly, the Task Force only evaluates the value of screening tests on healthy individuals at average risk for diseases or conditions. The 16 volunteer Task Force members come from primary care or preventive medicine—usually internal medicine, pediatrics and obstetrics-gynecology—and have special skills in evaluating evidence. They are often called “methodologists,” denoting this expertise. They are not clinical specialists. Indeed, a founding goal of the Task Force was to exclude clinical specialists who saw high-risk patients and might have already formed opinions about the value of screening tests.¹³ Task Force members serve one or more four-year terms, ensuring substantial turnover.¹³

From 1984 to 1998, Task Force members and their staff did their own literature and statistical reviews, but since 1998, outside groups do this background work. When the Task Force has completed its assessment, it assigns letter grades to screening tests. An “A” or a “B” means that a test is recommended, a “C” (which formerly meant no recommendation) means that the test may be appropriate for certain patients and should be discussed between provider and patient. A “D” discourages testing and an “I,” short for “insufficient,” means there is inadequate data to opine.¹⁴

The Task Force has used clinical epidemiology approaches to reach its conclusions. For example, concepts such as “lead-time bias” and “length-time bias” remind us that even though screening might identify disease earlier, if the patient’s ultimate outcome is the same, the apparent value of screening is misleading. The Task Force has also relied on other concepts, such as the causal pathway (later termed the analytic framework), to ensure that the desired outcome can be traced back to the screening test being studied.¹⁵ Another major emphasis of the Task Force has been to study the net health outcome of screening tests: not only their benefits but also the potential harms they cause. If screening saves lives but leads to substantial unnecessary invasive testing and worry, recommendations should take into account both of these outcomes.¹⁵ This consideration is especially important because those being screened are, by definition, healthy people, and doctors are supposed to “do no harm.”

CRC SCREENING IN THE EARLY 1990s

What is the history of cancer screening? Physicians in the 1800s believed cancers grew slowly and long stayed localized. Thus, finding lesions early, and removing them by surgery, would save lives from potentially deadly cancers. This led the American Society for the Control of Cancer (now the American Cancer Society), founded in 1913, to advocate for early detection of cancers. One cancer that exactly fit this model was cervical cancer, for which screening with Pap tests dramatically lowered both incidence and mortality beginning in the 1940s.¹⁶

Over the succeeding decades, the American Cancer Society (ACS) and other groups promoted early detection. As of the early 1990s, the ACS, the National Cancer Institute and the American College of Physicians advocated screening for colon cancer with either sigmoidoscopy (a test that visually examines the lower part of the colon) or Fecal Occult Blood Testing (that identifies invisible blood in the stool), even though there were no RCTs of either test.¹⁷ The stakes were high. Advocates who were quite certain, through their own clinical judgment and available data, that screening saved lives, even accused their methodologist colleagues on the Task Force of “killing” patients by waiting for more rigorous proof. (Of course, screening interventions that are prematurely approved and later rejected may cause harms and deaths as well.)

In its 1989 assessment, the Task Force disagreed with the other groups. It stated there was insufficient evidence to recommend either for or against CRC screening—even though screening seemed to make sense and might save lives.¹⁸ This disinclination to advocate screening tests that were popular, supported by anecdotal evidence and favored by experts, interest groups and professional societies, was emblematic of the Task Force. Superior evidence was all that mattered. Specialty societies who disagreed should try to conduct RCTs to help generate proof that screening actually worked. Indeed, there were several clinical trials of CRC screening underway at the time.

THE TASK FORCES CHANGES ITS MIND

By 1996, when the Task Force again addressed CRC screening, two trials of Fecal Occult Blood Testing (FOBT) had been published, showing up to a 33% reduction in mortality from colorectal cancer with regular testing.^19,20 To better understand the results, the Task Force had actually invited researchers from one of the trials to a meeting to detail their findings. Steven Woolf, then on the Task Force staff and later a member, said the meeting occurred “so the members of the Task Force who were really focused on taking trials apart and finding their design flaws would be able to ask them directly to satisfy themselves that the quality of the study was adequate to justify [a positive] recommendation” (Steven Woolf, M.D., personal communication).

Ultimately, the Task Force concluded there were now adequate controlled data showing that FOBT was indicated for individuals aged 50 and older. Yet even as the Task Force insisted on RCTs to countenance FOBT, it concurrently decided to also recommend sigmoidoscopy based on data it normally saw as insufficient: a retrospective case control study.²¹ This type of study compared two groups—in this case individuals who had received sigmoidoscopy versus those who had not—and controlled confounding variables to ascertain whether the intervention had worked. The study in question, which reported a mortality reduction from sigmoidoscopy as high as 70%,²² was deemed to be of such good quality that the Task Force awarded a “B” recommendation for both sigmoidoscopy and FOBT.

This decision demonstrated that even though the Task Force relies on a hierarchy of evidence, there is still a degree of interpretation that accompanies the analysis of data. Six years later, this point would become even more clear.

COLONOSCOPY AND MORE

When FOBT testing was employed, it was only the first step in the screening process. If positive, it needed to be followed by some type of visualization of the colon to try and discover what was causing the bleeding. One possibility was sigmoidoscopy but another was colonoscopy, which studied the entire colon. It was the finding and removal of precancerous adenomatous polyps or early malignancies during these tests that presumably actually lowered the mortality from colon cancer.²³ But there were no RCTs of colonoscopy.

The growing use of colonoscopy raised two important questions as the Task Force did its next assessment of CRC screening, which would come out in 2002. First, even though FOBT testing plus colonoscopy appeared to save lives, was it acceptable to recommend colonoscopy without formally testing it in an RCT? After all, the procedure was expensive and also carried substantial risks, including bleeding and perforation of the colon when polyps were removed. And second, if colonoscopy was routinely used after positive FOBT tests, and examined more of the colon than sigmoidoscopy, then shouldn’t colonoscopy be an approved screening test for CRC by itself?

By 2002, the ACS and several other organizations, believing there were adequate supportive data, were answering yes to both questions. And this time the Task Force agreed with them, giving a “A” recommendation that strongly recommended CRC screening for men and women aged 50 and older using either FOBT, sigmoidoscopy or colonoscopy. In the “Rationale” section of its recommendation, the Task Force explained why. “Efficacy of colonoscopy,” it wrote, “is supported by its integral role in trials of FOBT, extrapolation from sigmoidoscopy studies, limited case-control evidence, and the ability of colonoscopy to inspect the proximal colon”.²⁴

In other words, based on what was known about existing CRC screening and the presumed way that it lowered mortality, it made sense to the Task Force that colonoscopy by itself would also be effective. As Steven Woolf notes, “If you’ve established that testing using flexible sigmoidoscopy reduces mortality and now you’ve come along with an instrument with an even better ability to visualize—with better sensitivity and specificity—than the one you originally tested, then it is reasonable to infer that it too will reduce mortality even more effectively” (Steven Woolf, M.D., personal communication).

The Task Force knew that such a decision was potentially problematic. After all, it was assumptions about screening based on incomplete data that had generally led it to reject recommendations by specialists and other experts—such as the routine use of screening mammograms for women in their forties or prostate specific antigen (PSA) testing—during its periodic evaluations of these topics. In addition, it could be argued that the uncertainty of the evidence for colonoscopy increased the need for the Task Force to show it had net benefit before recommending it.¹⁵ That is, although colonoscopy might indeed be the most sensitive and specific CRC screening test, advocating it as a primary test might lead to more harms than merely doing a less risky sigmoidoscopy or doing a yearly FOBT to see if a colonoscopy was actually necessary. But the Task Force had carefully decided to use the data it had, as well as knowledge of the likely disease process, to make its decision. An RCT proving that colonoscopy saved lives would have been preferable, but was deemed unnecessary.

Over time, as it assessed CRC screening, the Task Force increasingly saw its role as broader than merely evaluating the existing data. A crucial issue had become adherence. CRC screening, it believed, not only saved lives but was cost-effective on the population level. However, there were not enough randomized data to fully evaluate the growing number of available screening tests.^20,25 Thus, in 2008, rather than limiting its repeat “A” recommendation to specific screening tests, it gave an “A” to CRC screening in general and noted in its “Clinical Considerations” section that while some of the screening tests were superior to others, “efforts to reduce colon cancer deaths should focus on implementation of strategies that maximize the number of individuals who get screening of some type.” Individual clinicians and patients could use “shared decision-making” to choose a particular screening test.²⁶

Of note, in 2008, for the first time, the Task Force recommendation relied in part on microsimulation modeling—a computer-based analytical methodology that used criteria, such as known incidence of disease, stage distribution and survival—to predict events and outcomes.^1,27,28 This major step, which used projected data as opposed to actual data, was an example of what was sometimes termed “evidence-informed” rather than “evidence-based” medicine.²⁷

The emphasis on adherence over definitive data accelerated even more in 2016, when, in conjunction with another “A” recommendation for persons 50 and older, the Task Force stated that the best screening test was “the one that gets performed”.¹ In its Clinical Considerations section, the Task Force not only included FOBT, sigmoidoscopy and colonoscopy (which still had not been evaluated by an RCT), but four other screening modalities—fecal immuno-chemical testing [FIT], computed tomography colonography, FIT DNA testing, and the combination of sigmoidoscopy and annual FIT testing—for which there were also no RCT data.²⁹ Support for colonoscopy was enhanced by a 2012 study showing that removal of adenomatous polyps prevented deaths from colorectal cancer.³⁰ But in promoting a wide array of screening modalities, the Task Force once again relied substantially on modeling studies that estimated life-years gained from the various strategies.^1,31

The 2016 guidelines generated substantial criticism. For example, a 2017 article by Norwegian physician Michael Bretthauer and colleagues argued that giving equal approval to the five screening tests for which there were no randomized data was “contrary to the principles of evidence-based medicine”.³² Whether or not one agreed with Bretthauer about the definition of EBM, the extrapolation process that had begun in 2002 with the inclusion of screening colonoscopy had expanded substantially by 2016. Confidence in CRC screening in general, and a desire to maximize its diffusion, had led to a more inclusive embrace of certain technologies than the available data conclusively demonstrated.

SCREENING BLACK PATIENTS

Just as the Task Force was carefully expanding its recommendations, it was becoming apparent that one group in particular was not benefitting enough from CRC screening: Black Americans. Data from the late 1990s showed that Black patients had a 12.3 % higher rate of colon cancer than whites and were 16 % less likely to survive five years.³³ Researchers listed several explanations for this phenomenon. Black patients were less likely to undergo screening, they were diagnosed with colon cancer at later stages and they were less likely to receive adjuvant chemotherapy and radiation.³⁴ Among the explanations given for these discrepancies was that Black patients had worse insurance and worse access to care; the legacy of racism in America; a disinclination of some Black patients to pursue aggressive medical care; and the fact that they had higher rates of cancer in the proximal colon, which was harder to diagnose.^35–39

In response to this problem, there was one possible solution that everyone could agree on: medical providers needed to pay special attention to Black patients, and provide them with social supports, to improve their rates of screening. As the Task Force wrote in 2016, “Efforts are needed to ensure that at-risk populations receive recommended screening, follow-up, and treatment”.²⁹

But there was another possible strategy: starting to screen Black patients earlier, perhaps at age 45. A 2001 study had reported nearly twice as high an incidence of colon cancer in Blacks under the age of 50.⁴⁰ In 2005, the American College of Gastroenterology stated that current data “would favor modification of the CRC screening guidelines for African Americans to begin screening at the age of 45 yr rather than 50 yr”.³⁴ The American Society for Gastrointestinal Endoscopy followed suit in 2010.⁴¹ In 2012, the American College of Physicians recommended that Black patients be screened starting at age 40 due to their higher incidence.⁴² Among the data promoting these decisions was a modeling study from 2009 that found that screening Black patients earlier added additional life-years and could “play a significant role in reducing disparities between blacks and whites”.⁴³ Then, in 2015, a review of the Surveillance, Epidemiology and End Results (SEER) database found that while the incidence of colorectal cancer for whites began to increase at age 47, it did so at age 43 for Black patients.⁴⁴ Finally, in 2017, the U.S. Multi-Society Task Force on colorectal cancer weighed in, advocating screening at age 45 for Black patients.⁴⁵

Despite these data and the new recommendations, the Task Force continued to not create a separate category for Black patients. The major issues, according to the “Clinical Considerations” section of its 2016 report, were the inequities in screening, diagnostic follow-up and treatment in Black populations. In addition, the only data suggesting that earlier screening would be beneficial were the above modeling studies—as opposed to clinical trials. In this instance, therefore, largely lacking evidence of any net benefit of colonoscopies under 50, the Task Force chose to not create a separate recommendation for Black patients but rather to advocate for rectifying their substandard care.

The Task Force’s decision is interesting to analyze in light of some of its other recommendations regarding CRC screening. As we have seen, the Task Force extrapolated from FOBT and sigmoidoscopy data to approve screening colonoscopy despite a lack of randomized trials. It then, based on modeling, also recommended several other less studied screening tests as a way to improve adherence.

One might have imagined a similar dynamic at play here. Colonoscopy saves lives by finding polyps earlier and removing them. Faced with a complicated sociomedical problem of improving access for Black patients, the Task Force might have joined other groups in arguing that it also made sense to find and remove potentially fatal polyps and cancers earlier among Black patients—a strategy that almost certainly would have saved some lives. But instead, the Task Force said no, thus mirroring its early 1990s disinclination to approve seemingly logical colorectal cancer screening tests before there were adequate data, paying close attention to not only the potential benefits but also the potential harms. Perhaps the conservatism here also reflected a disinclination to potentially cause unnecessary harms to a population—Black patients—who had historically been victimized in unethical, at times racist, human experiments.⁴⁶ “If we don’t know whether screening works [for those under 50],” David Ransohoff argues, “then we don’t know whether screening works and it doesn’t matter if you have a higher prevalence” (David Ransohoff, M.D., personal communication). (The Task Force’s recent 2021 CRC screening recommendation has indirectly “resolved” the debate. By advocating screening all persons aged 45 and older, Black patients will now have access to earlier testing, although not preferentially).

CONCLUSION

The history of colorectal cancer screening provides an excellent opportunity to revisit the efforts of the U.S. Preventive Services Task Force to bring EBM to clinical practice. The Task Force has always had its critics, but it is reasonable to argue that throughout its existence, it has dutifully striven to digest an enormous amount of evidence, generate fair conclusions based on that evidence and then disseminate its findings to benefit patients, health professionals and society.^47–49 The Task Force has also made major efforts in recent years to improve its transparency and elicit feedback as it generates its conclusions.⁵⁰ In addition, it includes the recommendations of other groups as part of its reports to try to lower confusion among its readers.

To be sure, the Task Force’s major focus has always been the assessment—and ranking—of different types of evidence. Nevertheless, as David Sackett originally stated, EBM is not cookbook medicine. Sackett was referring to the need for physicians to include clinical expertise and patient values when evaluating specific interventions, but what other factors has the Task Force considered in making its judgments about screening? This paper offers a window into the process.

As we have seen, Task Force decision-making is a dynamic process. At times, it has insisted firmly on the need for definitive RCTs. At other times, when there were inadequate data, it has drawn conclusions through other forms of evidence, such as extrapolation. At times, it has shown great faith in modeling studies, even using them to promote what we might term “social” causes, such as improving adherence to CRC screening. At other times, as with lowering the screening age for Black patients, it has viewed the conclusions of modeling studies as inadequate and waited for “better” data. In a sense, the Task Force’s judgments have been as much cognitive attempts at crafting the most logical and coherent recommendations for clinical practice as statistical analyses of the data.

This conclusion is not meant to be a criticism of the Task Force as somehow arbitrary or inconsistent. That evaluating evidence often involves subjective decision-making will come as no surprise to its members—even though they are the best evaluators of data that we have. But with the frequent public squabbles about what data do and do not “show,” and the growing connection of recommendations to funding decisions, the perception often exists that groups like the Task Force are supposed to reach “objective” conclusions. General internists—and their patients—should realize that EBM recommendations are not definitive answers but rather educated and thoughtful data-based advice. Scientific evaluation will always rely not only on methodological sophistication but also a combination of intellectual, cognitive and social processes.^51–54