NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Woloshin S, Schwartz LM, Welch HG. Know Your Chances: Understanding Health Statistics. Berkeley (CA): University of California Press; 2008.

Cover of Know Your Chances

Know Your Chances: Understanding Health Statistics.

Show details

Chapter 5Not All Benefits Are Equal

Understand the Outcome

The Zocor ad introduced in chapter 4 mentions two benefits for people who have high cholesterol and heart disease: the drug lowers cholesterol levels, and it reduces the chance of death from a heart attack. The drug has other benefits as well: it reduces the chance of a nonfatal heart attack, and it lowers the chance of developing a weakened heart—a condition known as heart failure—which makes sense, since fewer heart attacks mean less weakening.

All the things Zocor affects—cholesterol levels, the chance of fatal and nonfatal heart attacks, and heart failure—are called outcomes. As you’ve learned, the benefit of drugs or other interventions is measured in terms of how they affect outcomes. In chapter 4, we listed some questions that you should ask whenever you hear a message about how an action can reduce your risk. The first question to ask is this: “Reduced risk of what?” The answer is the outcome.

Only when you understand the outcome under consideration does it make sense to ask the second question: “How big is the risk reduction?” Without a clear picture of the seriousness or importance of the outcome, you can’t decide if the risk reduction is big enough to make the action worthwhile. In fact, if you don’t care about the outcome, there’s no reason to even bother learning the size of the benefit.

So how do you decide if you care about the outcome? The following illustration organizes outcomes according to their direct impact on you. We call it the pyramid of benefit. At the bottom are things that don’t have a direct impact on you—for example, laboratory measurements, blood tests, and X-rays. You don’t directly experience or feel these outcomes; for instance, you don’t feel any different if your cholesterol level is really low or really high. On the pyramid, these kinds of outcomes are called surrogate outcomes.

As you move up the pyramid, you encounter patient outcomes, where the impact of the outcomes becomes increasingly direct: you clearly feel symptoms such as pain and nausea, for example, and you directly experience events such as being hospitalized, requiring nursing home care, or needing an operation. And since it is hard to think of anything that has a more direct impact on you, death appears at the top of the pyramid. Because the pyramid may be useful in helping you decide which outcomes you care about, the following sections examine it in some detail.

Image ucalpkycf9.jpg

Better Test Results

Many medical interventions change surrogate outcomes, such as laboratory measurements of cholesterol levels or X-ray results that measure bone density. Since you cannot directly feel these surrogate outcomes, why should you care about them? The reason is that these outcomes often “stand in” for, or represent, patient outcomes higher on the pyramid. A surrogate is something that stands in for something else.

Unfortunately, surrogate outcomes often do a poor job of “standing in” for patient outcomes. Keep this in mind! Even if a drug improves a surrogate outcome, it doesn’t mean that the drug will improve a corresponding patient outcome. Judging benefit based solely on a surrogate outcome requires a big leap of faith.

As an example, consider a drug that has been demonstrated to lower cholesterol levels (a surrogate outcome). A host of medical studies have shown that lower cholesterol levels are linked to fewer heart attacks (a patient outcome). Does this mean that the drug will reduce the number of heart attacks? It may—but it may not. Improving a surrogate outcome does not automatically mean improving a patient outcome. (See the Learn More box on page 58.)

Box Icon

Box

Learn More. There are many examples of beneficial changes in surrogate outcomes that fail to translate into beneficial changes in patient outcomes. Let’s look at the famous example of clofibrate, a cholesterol-lowering drug that actually increased (more...)

Consider a health message, adapted from a real ad for an osteoporosis drug, claiming that “drug X improves bone density by 35 percent.” (Osteoporosis means thinning bones.) This improvement may sound pretty encouraging. But let’s take a closer look. To start, ask yourself, “What is the outcome?” The answer is greater bone mineral density. Unless you know what bone density means, the 35 percent statistic won’t matter to you. Bone density is a way of gauging how strong bones are. But you don’t feel bone density—it’s a surrogate outcome. The only reason we care about bone density is because weak, thin bones are more apt to fracture. And bone fractures—something people directly experience—are important: they hurt, and they can leave people disabled. For the elderly, hip fractures in particular are often the first step toward institutionalization or even death.

But what if drug X improved your bone density (measured by a T-score) but did not lower your chance of a hip fracture, the most disabling kind of bone fracture? This is not so far-fetched. Lots of things other than bone density affect your chance of fracturing a bone, including factors that increase your chance of falling, such as poor vision, poor balance, or even how your house is set up (lots of clutter, throw rugs). And increased bone density is not the only ingredient of bone strength—for example, bone architecture (the underlying structure of the bone) may matter even more. As it turns out, although many drugs improve bone density, few have been shown to reduce hip fractures.

If drug X works only on the surrogate outcome (bone density), but this benefit does not translate into the corresponding patient outcome (fewer hip fractures), you would be wise to be hesitant about exposing yourself to the cost, inconvenience, and potential side effects of the drug. This would be true no matter how big an effect drug X has on the surrogate outcome.

You cannot reliably assume that improving a surrogate outcome will translate into better health. Nevertheless, you will hear about surrogate outcomes all the time (especially in drug ads), because they are easier and faster to measure than patient outcomes: a drug may lower cholesterol levels right away, but it will take years to see whether the drug really results in fewer heart attacks. If you hear about a medical intervention that improves a surrogate outcome, ask this question: “Has the intervention also been shown to have a beneficial effect on what people experience or feel?”

QUIZ

A study in a major medical journal reports that a new drug can shrink Liver tumors. What is the most important additional information you would want to know?

  1. How much the tumors shrink
  2. Whether people feel better or live longer

The correct answer is b. Tumor shrinkage is a surrogate outcome. It may translate into longer, better lives—but it may not. Unfortunately, we have many examples of cancer drugs that shrink tumors but do not extend life (some even shorten it). Surrogate outcomes matter only to the extent that they improve the corresponding patient outcome (the outcome they stand in for). So knowing how much the drug shrinks the tumors isn’t really important—unless the drug makes people feel better or live longer.

Now you know why we consider surrogate outcomes to be less important than patient outcomes.

Fewer Symptoms of Disease

The next step on the pyramid addresses the benefit of fewer symptoms of disease. Symptoms are the disease-related sensations people feel. The most familiar symptom is pain: headache, back pain, joint pain, chest pain. Many other common symptoms can be very unpleasant though not painful: upset stomach, shortness of breath, lightheadedness, and runny nose, for instance.

Most people do not need to think long and hard about the importance of interventions that make them feel better when they are sick. That’s why we believe that a reduction of symptoms is a very important outcome. For most people, symptom relief matters so much that they are eager to get to the next questions and ask for the numbers: “How big is the benefit? That is, how likely is it that I will feel better if I take this action? And how much better will I feel?”

In fact, some people wouldn’t even bother asking about the numbers. People with symptoms can often see for themselves whether the intervention works, by trying it. For example, if your headache doesn’t improve after taking a headache medicine, you may decide that the pills don’t work. Alternatively, if the headache goes away completely in twenty minutes, you’ll probably decide that the drug is a winner.

QUIZ

Mr. Smith started taking a drug for restless legs. After 12 weeks, he says that his legs feel much better. Can you be sure that the improvement is a result of the drug?

  1. Yes
  2. No

The correct answer is b. Symptom improvement is not a foolproof test of benefit. Some people feel better just because they do something. You may have heard of the placebo effect: people sometimes experience a benefit even when they take an ineffective sugar pill or when they receive a faked surgery. And some symptoms, by their very nature, wax and wane spontaneously. People with back pain know this quite well: on some days, their back feels great; on other days, it feels awful. These two factors—the placebo effect and spontaneous improvement—can lead people to judge an intervention as being beneficial when in fact it is not. For these reasons, the most trustworthy test of an intervention for current symptoms is a randomized trial—a true experiment, in which people are randomly given either the treatment or a placebo and then undergo a standardized symptom assessment. If the treatment works, the people who were randomly chosen to receive it will, on average, do better than those randomly chosen to receive the placebo.

Deciding whether a treatment works by trying it is possible, of course, only when you have symptoms—this strategy doesn’t work with interventions that are designed to reduce your risk of some future event. If you take Zocor, for example, you won’t feel your “reduced risk” of a future fatal heart attack. The only way to gauge the benefit of interventions that reduce a future risk is to know the numbers: your starting and modified risks, as outlined in chapter 4.

Learn More

When Symptom Improvement Isn't Enough

Requip—the first drug to treat restless legs syndrome—is a great example of how hard it can be to judge whether a drug is the reason you feel better. In the largest study of the drug, about 200 people with moderate to severe restless legs syndrome were given Requip.2 After 12 weeks, about 70 percent (70 out of 100) of these people had substantial improvement. (This is the modified risk.) Does this mean that Requip works? No. Unless you can compare this improvement to what would have happened without Requip, you can’t understand whether there is a benefit. That’s why you need a comparison group.

The comparison group in this study consisted of 200 people with moderate to severe restless legs syndrome who were given a placebo. Interestingly, 55 percent (55 out of 100) of the placebo group also had substantial improvement. (This is the starting risk.) Based on this comparison, the benefit of taking Requip is the difference between 55 percent improvement and 70 percent improvement. In other words, given that so many people got better without any treatment, we can conclude that only 15 percent (15 out of 100) actually felt better because of the drug. When there is such a big placebo effect, it is really hard to know whether feeling better is in fact a result of the drug. Unfortunately, this can lead many people to continue to take a medication that isn’t really helping them.

Fewer Deaths from Disease

Because reducing the number of deaths from disease is a very important outcome, it appears at the top of the pyramid. There isn’t a lot of ambiguity here—death is really important to most people. So interventions that reduce the number of deaths almost always matter (a lot) to people. But you should still go ahead and find out the numbers. You might be surprised: you might not care so much about an intervention that reduces a very rare cause of death, or even one that reduces a common cause of death by a very, very small amount.

Learn More

Fewer Diagnoses of Disease: How Important?

The pyramid of benefit shows three categories of benefits that can result from medical interventions: fewer deaths, fewer symptoms, and better test results. But there is another category of benefit to consider: fewer diagnoses of disease. Some interventions reduce the chance that you will be diagnosed with a disease.

On the face of it, reducing your chance of being diagnosed with a disease sounds like a very desirable benefit—and it can be. But recognizing how important this benefit is can be surprisingly tricky. To understand why, try to decide where it fits on the pyramid of benefit.

Do fewer diagnoses of disease always mean fewer deaths? Certainly not. For most cancers and other serious diseases, diagnosis does not equal death.

Nor do fewer diagnoses always mean fewer symptoms of disease. That’s because many diseases, especially in their milder forms, have no symptoms to begin with. For example, except in the most extreme cases, high blood pressure has no symptoms. Nor does mild type 2 diabetes, anemia, or kidney disease. These diagnoses are made by doing blood tests.

So sometimes fewer diagnoses simply mean better test results. But in those cases, the outcomes may not be very important. That’s because test results are surrogate outcomes—and as we’ve discussed, surrogate outcomes are not always closely linked to patient outcomes. Not everyone with mildly abnormal test results develops severe abnormalities or symptoms. So preventing diseases that are diagnosed only by tests will not translate into a tangible benefit for everyone. The link between better test results and fewer symptoms is tenuous—and there may be no link between better test results and fewer deaths.

The point is that fewer diagnoses of disease can have a range of meanings. Some interventions that result in fewer diagnoses matter more than others. We think that, in general, the benefit of fewer diagnoses of diseases with symptoms is more important than the benefit of fewer diagnoses of diseases defined just by test results. To help yourself decide how much this benefit matters, ask: “Do fewer diagnoses mean that I will live longer? Or feel better? Or feel better longer? Or just have a better-looking medical record?”

We should also acknowledge that, in some cases, reducing the risk of death is not always the most desirable effect. For someone with terminal cancer or end-stage Alzheimer’s, minimizing suffering—symptoms—may be more important than prolonging life.

In this chapter, we showed you how the pyramid of benefit can help you decide how much to care. Of course, the less you care about the outcome, the less important it is to bother thinking about the size of the benefit.

Pay attention to the outcome that is addressed in any health message—it may not be the one you care about most. When you hear about how a drug affects a blood test result, such as cholesterol or bone density scores, remember to keep your eyes on the prize: cholesterol, other lab tests, X-rays, and so on may matter. But they may not. Do people who take the drug live longer? Do they feel better? Or does the drug just make their test results look better? What really matters to you is how these things translate into how you feel, how they affect your chance of living the way you want to live, and whether they affect your chance of dying.

Copyright © 2008, The Regents of the University of California.

Know Your Chances: Understanding Health Statistics is hereby licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported license, which permits copying, distribution, and transmission of the work, provided the original work is properly cited, not used for commercial purposes, nor is altered or transformed.

Bookshelf ID: NBK126171

Views

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...