Moneyball, Gambling and the New Cholesterol Guidelines

“Lies, damn lies and statistics.” The old adage is emblematic of people’s distrust of statistics, yet in recent years advanced statistics have proven useful over traditional approaches in areas as varied as Nate Silver’s work in political polling and advanced statistics in sports. In November 2013, new cholesterol guidelines, initially developed by an expert NIH panel and then endorsed and published jointly by the American College of Cardiology and the American Heart Association (ACC/AHA), took a few steps toward applying more advanced statistical techniques to medical practice.¹

The fundamental change is that the ACC/AHA cholesterol guidelines base their recommendations on the best available information on an individual patient’s chance of benefitting from treatment. To achieve this, they noted that we need to stop focusing so much on LDL cholesterol levels, and focus more on our patient’s overall risk of having a heart attack or stroke. For primary prevention, the new guidelines propose that a statin be recommended once a person’s 10-year risk of having a heart attack or stroke is above 7.5%, using a CV risk calculator developed by the original NIH expert panel. Noting that multiple clinical trials have failed to find a significant benefit from adding a second lipid lowering medicine to a statin, the ACC/AHA cholesterol guidelines generally recommend no additional cholesterol medications for those that tolerate a statin, even if LDL remains elevated.

These guidelines have triggered a cacophony of controversy, with some cholesterol specialists suggesting that the new guidelines will lead to serious under-treatment of high cholesterol and lost lives, and others suggesting that they will lead to over-treatment of millions of Americans.^2,3 The net result – utter confusion.

Controversies in medicine have become a dime a dozen, but this medical controversy has a unique aspect to it. For the first time that I know of, a major medical guideline considered a statistically powerful approach for individualizing estimates for the odds of a patient benefitting from a treatment.⁴ Producing information on the odds of an individual patient benefitting from a specific treatment offers exciting opportunities, but it also reveals three unwanted truths: 1) medical decisions represent gambles, 2) the chances of benefit and harm exist along a continuum, and 3) these odds are rarely precisely known.^5,6 These three facts are not new; they are inherent to medical decisions. However, these distasteful realities are easy to ignore under the traditional guideline paradigm, which does not even try to quantify the odds of benefits and harms for individual patients. Ignoring them does not make them go away, though. To better understand these principles, it might help to start by reviewing a recent lesson from baseball.

Why batting average and LDL treatment goals aren’t important anymore

In his 2003 bestseller Moneyball, Michael Lewis tells the tale of frustrated Oakland A’s general manager, Billy Beane, later portrayed by Brad Pitt in the movie. Billy Beane is a driven man with a big problem. He desperately wants to win, but big market teams like the Red Sox and Yankees tend to get better players because they can outspend him 2–3 fold. Willing to try almost anything, he begins to listen to the advice of statistical experts -- nerds that most “baseball experts” find either amusing or irritating, and often both. After all, they endlessly play with numbers and look about as athletic as tee-ball players. Further, they confidently say outrageous things, like managers should ignore a batter’s RBIs and a pitcher’s win-loss record when making decisions about the value of players.

Most sports fans know the rest for the story (as well as many who see Brad Pitt movies for other reasons). Billy Beane puts together excellent teams on a budget, competing with the richest teams in baseball. I smile when I hear people say that he did this “against all odds”, because Beane did it by understanding the odds better than most others. The math nerds did better than “baseball experts” by using regression analysis to determine which bits of information are the most useful, such as what traits of a batter best help estimate how many runs one batter, compared to others, will produce for a team. Back in the 1970’s, these techniques demonstrated that managers should abandon the traditional metric of Batting Average in favor of the more useful statistic, On Base Percentage, which accounts for walks as well as hits in evaluating a batter’s chance of getting on base.

What does this have to do with the cholesterol guidelines? Similar to batting average, colleagues and I pointed out in 2006 that once you knew a person’s overall cardiac risk, a person’s LDL was completely unimportant in estimating how much a statin reduces a person’s risk of a heart attack.⁷ In 2010, we used more advanced statistical methods to demonstrate how to best estimate an individual’s potential benefits from different doses of a statin by using a person’s 5–10 year risk of having a heart attack. We were careful to point out, however, that although these estimates are better, they still have non-trivial levels of uncertainty.⁴ Further, even if the odds were known precisely, there is no magical risk level that can divine when the potential benefits of a treatment outweigh the risks.^5,6,8

Medical decisions are gambles with uncertain odds

Let's return to Moneyball. When compared to the old statistics, advanced baseball statistics gave Billy Beane a better way to estimate a player’s future run production. Knowing a player’s advanced stats simply gave Beane better information to make better gambles, but it did not give him the superpower of omniscience. Knowing the precise outcome for any individual player in advance is impossible, whether it’s their next at bat or next season’s stats. Further, in making good decisions Beane needs to consider much more than a player’s estimated run production. He has to consider “team needs” and the difference in player “market prices”, their defensive skills and their likely impact on teammates, both positive (eg, leadership) and negative (eg, behaving like a diva). Most of these factors are even harder to predict than their potential run production. Having better information can aid decision-making, but does not substitute for it. Advanced baseball statistics have quite appropriately not resulted in strict decision rules, such as 10 points of the Runs Created stat is worth $1 million more per player per season. A manager would be foolish to ignore advanced statistics and equally foolish to slavishly let them dictate their decisions.

Unfortunately, most guideline committees continue to refuse to accept this basic fact of life. The NIH expert panel and ACC/AHA should be commended for recommending abandoning LDL targets, but they should have also refrained from setting, or at least appearing to set, rigid thresholds for treatment decisions. Clinical guidelines can be highly useful, but guidelines should focus more on providing guidance and perspective, and focus less on setting rigid rules.⁹

Although medical decisions and patient outcomes are often categorical – treat vs. don’t treat, and died vs. lived – the odds of a treatment changing outcomes lie along a continuum (almost 0% to almost 100%), and there is no thin bright line that determines when the odds are “good enough”.^5,6 This is not limited to baseball and medicine; most choices in our lives represent gambles. Fortunately for our peace of mind, most of our everyday gambles have potential losses and winnings that are fairly trivial. For example, “Should I take my umbrella with me when there is a 10% chance of rain and I only have a 1 minute walk from the subway station to my office?” In this instance, I have about a 1 in 10 chance of losing – getting wet in my 1 minute walk – and I would have to pay the cost of carrying my umbrella with me to avoid this risk. (For the record, I would never take my umbrella with these odds but my wife would always take hers, but then, I have a much greater chance of losing my umbrella, so my gamble is worse than hers.) Sometimes the wager and stakes are profound: “Should I undergo a brutally aggressive chemotherapy treatment that reduces my risk of dying from cancer in the next 10-years from 10 in 100 to 8 in 100?” Both the wager and stakes are now staggeringly high, with a 1 in 50 chance of chemotherapy preventing me from dying from cancer, and a 49 in 50 chance of going through hell for nothing.

Weighing the risks and benefits of taking a statin for primary prevention

So, let’s break down the gamble for someone with a 10-year heart attack risk of 7.5%, the point at which the new guidelines recommend starting a statin. Clinical trials suggest that a standard statin dose reduces heart attacks for primary prevention by about 35%–40%,^4,10 but this does not mean that it decreases your chance of having a heart attack by 35–40 in 100; it is a relative reduction; a statin will reduce this individual’s absolute chance of having a heart attack in the next 10 years from roughly 7–8 in 100 to roughly 4 –5 in 100 people (ie, reduces the person’s 7.5% risk by 35%–40%). So for every 100 individuals at this risk level that take a statin every day for the next 10 years, about 3 will win by avoiding a heart attack. Is this a good bet? Just like in the baseball example, we need to consider other aspects of this gamble, such as, “If I get a heart attack, how bad is it likely to be?”, and “What are the risks of side effects and harms from taking a statin for 10-years?”

As it happens, most risk tools peg my 10-year heart attack risk to be right at the proposed treatment threshold of 7% to 8%. After thinking things over, I have decided to not start a statin…. yet. Why? First, my risk is probably somewhat lower than predicted given information that I know that is not in the risk calculator, such as my diet and exercise habits. Second, even if my estimated 3 in 100 chance of avoiding a heart attack is accurate, in my age group most heart attacks are minor. Third, I do not plan to wait 10-years before starting a statin, and most of these potential heart attacks prevented can still be prevented if I start a statin 4–5 years from now. Yes, I could lose this wager; I could have a major heart attack in the coming year that a statin would have had a chance of preventing. But there are no risk-free choices in life, just gambles. For me, my relatively small chance of benefitting is not yet great enough for me to want to start a statin at my age (58), especially since we have so little information on statin’s long-term safety. We know that statins are pretty safe over 5–10 years, but I have a very good chance of living another 25 years (my parents are pretty spry in their 80s), and there are some recent hints that in the long-term statins might have negative effects beyond what we currently believe. Of course, these concerns would be unimportant if my chances of benefitting were much higher.

The New Cholesterol Guidelines

The reaction by some medical specialists to the new statistical approach to estimating the benefits of taking a statin has been to call for a retreat to the old style of making guidelines.^2,3 This involves basing recommendations on LDL levels or on the average results in clinical trials without much effort to estimate the odds of individual’s benefitting and being harmed by the treatment. To my mind this is the worst approach – a “see no evil, hear no evil” approach. It pretends that if you do not try to identify which individuals benefit more than others that the problem of medical decisions being gambles with uncertain odds goes away.^4,11

But I also object to the ACC/AHA cholesterol guidelines idea that when a patient’s 10-year risk is above 7.5% that a statin should be recommended. It uses a better approach than the old LDL target approach, but ignores the inherent imprecision of estimates of an individual’s chances of benefit and harm and dismisses individual patient preferences and circumstances.^5,6,8 They could have also provided greater provider/patient discretion at the other end of the spectrum. I agree that we should not routinely be adding a second lipid agent to achieve LDL targets, but in very high risk individuals, such as a middle age vasculopath, I would certainly discuss with them the pros and cons of taking a second agent. I would let them know we do not have grade A evidence for whether this helps, but there is some grade B evidence that it could result in a substantial absolute risk reduction, and see if the patient feels that it is worth the gamble. But note – the grade B evidence for this potential for substantial benefit is due to an unusually high CV risk; not because their LDL is not “at goal”. Most patients who start at high risk, are no longer at high risk once they are on a statin, appropriate blood pressure medications, daily aspirin and a Mediterranean diet, which evidence suggests reduces CV risk by over 70%.¹²

Many decisions (gambles) in medicine, and in life, lie in a grey zone in which the right decision is entirely one of personal preference, and guidelines need to start to recognize this reality. In response to criticisms in the media, a couple of guideline panel members have suggested that the 7.5% risk threshold was always meant as a point to start discussions with patients, not as a point that statins should automatically be given. If this is the case, the committee should make this point much more explicit in a revision of the guidelines (“shared decision-making recommended if 10-year risk is between 7.5% and 12.5%. This is especially important since guidelines are often converted as is into quality measures. Without the ACC/AHA cholesterol guidelines being more explicit on this point, there is a risk that their guidelines could result in doctors and health plans being evaluated based on the proportion of their patients with ≥ 7.5% heart attack risk who are on statins.^5,13

I hope that the new cholesterol guidelines start a trend for more individualized guidelines. Such approaches are feasible and can often reduce costs while improving outcomes.^4,14–15 We can start by insisting that guidelines try to follow three basic rules: 1. Whenever possible, they should explicitly consider whether what we are asking our patients to do is worth the expected absolute risk reduction (ARR), 2) consider how ARR varies across the population, especially by an individual’s risk if not treated, and 3) include a grey zone, in which a treatment is a matter of patient preference.^5,6,14,15 The new ACC guidelines fail on all three counts, but then so do almost all current guidelines.

It seems inevitable that eventually some form of using advanced statistics to estimate potential benefits and harms for individual patients will become routine in medical practice, but it is unlikely to be a quick or easy transition.^2,3,15 In baseball, the traditional “baseball experts” held back the widespread adoption of advanced statistics for decades. Most often the traditionalists pointed out the deficiencies of the new advance stats – “baseball is more than numbers” or “they predicted that Torres would do better than Jones, but Jones ended up having a much better year”. Their statements were true and irrelevant. Yes, the new approach is not magic, but it is demonstrably better. The question now is how long will medicine’s traditionalists succeed is promoting older, less accurate statistics and decision rules by pointing out that the new statistical methods do not resolve an unwanted reality – medical decisions are gambles and thoughtful decision-makers must make decisions based on imprecise information. We would be foolish to ignore this better information, but equally foolish to slavishly let it dictate our decisions.