NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Woloshin S, Schwartz LM, Welch HG. Know Your Chances: Understanding Health Statistics. Berkeley (CA): University of California Press; 2008.

Cover of Know Your Chances

Know Your Chances: Understanding Health Statistics.

Show details

Chapter 8Beware of Exaggerated Importance

Health messages—whether advertising, public service announcements, or media reports—often exaggerate their own importance. They may inflate the size of the health problems they present, and they may overstate the benefits of the course of action they promote. When you think about it, it’s pretty easy to understand why.

Advertising is all about putting products in the best possible light in order to increase sales and therefore profits. Drug ads are intended to persuade people to take the drugs. As marketing strategy, exaggerating the benefit of a drug and downplaying side effects make sense.

The same logic applies to public service announcements, in which an organization (even with the most altruistic motives) is marketing a behavior: get a flu shot, know your cholesterol level, get screened for breast cancer. Once again, overstatement can help persuade you to do what the organization wants.

Finally, there’s no shortage of exaggeration in the media. News reports are especially good vehicles for exaggeration because so many people and organizations stand to gain financially and professionally from being associated with exciting news. Strongly favorable news coverage spurs sales for drug and device manufacturers, which in turn rewards their investors. Being associated with widely reported “breakthroughs” advances the careers of scientists and is a plus for their sponsors. Good publicity helps health care providers sign up patients and bolsters the causes of patient advocacy groups by attracting donors. Finally, media outlets themselves benefit from attention-getting news coverage: compelling stories sell papers and help journalists get their work on page one. As these forces converge, a self-reinforcing cycle of exaggeration is probably inevitable.

So what can you do? Our advice is to be a healthy skeptic. That doesn’t mean you have to be a cynic, simply disbelieving all the health messages you hear. Instead, it means approaching messages critically: looking out for—and seeing through—common tactics used to exaggerate the importance of health problems or actions you can take to address them. These tactics include emphasizing unimportant outcomes, avoiding numbers, or presenting statistics in ways that make them seem more important than they really are.

These two basic steps, which we’ve emphasized throughout the book, will help you to think clearly and critically about a health message:

Step 1. Be clear that the outcome matters to you. To begin deciphering a health message, you need to determine which outcome is being considered. If you don’t care about that outcome, then it makes no sense to act on—or think about acting on—the message. For example, you may hear that millions of Americans have prehypertension—that is, their blood pressure is at the high end of the range considered normal. Should you worry about whether you have prehypertension or take some kind of treatment for it? We would say no unless you are shown convincing evidence that this condition matters—that the people with prehypertension do worse in some tangible way or do better if treated.

Step 2. Get the numbers. The second step is to look critically at all the alarming numbers or dramatic stories that are being used to grab your attention. Consider the following news story leads:

“Mrs. Jones (shown herewith her three young children) was diagnosed with Liver cancer in 2003. She took drug X, and her cancer melted away.”

“Liver cancer will strike almost 19,000 people this year.”

“A new study has found that drug Y reduces deaths from liver cancer by 67 percent!”

“A new study has found that drug Z reduces liver cancer deaths from 3 in 100,000 to 1 in 100,000.”

Which stories grab your attention? Everybody responds to a compelling anecdote (the first lead). For most of us, an ounce of emotional content is worth a pound of data.

Big numbers also catch your attention (the second lead), as do large relative changes like the 67 percent relative change in liver cancer deaths (the third lead). This is a common tactic, which can leave you with an exaggerated sense of how well treatment works.

And, of course, a dry report of the starting and modified risks (the fourth lead) is both less impressive than the big numbers and much less compelling than the story about a person with young children whose life was “saved” by an experimental drug.

But, as you know by now, you need the data. Without the numbers, it’s hard to tell how big the risk really is, how big the benefit might be, or how much it matters to you. The only way to assess the importance of anecdotes (“Smith says he lowered his cholesterol by eating garlic”) and qualitative claims (“mammograms save lives” or “statins reduce heart disease”) is to examine quantitative data—specifically, the frequency of an outcome among people who receive or do not receive a treatment (the starting and modified risks).

Our advice can really be summarized in six words: know the outcome; get the numbers! You apply this principle regularly in daily life. How you react to news that taxes are increasing, rents in your building are going up, or salaries in your firm will rise depends entirely on the size of the increase. When it comes to money, it’s hard to imagine anyone who would hear the phrase “going up” and not want to know what the real numbers are. But, surprisingly, many people don’t demand the same kind of complete information when they hear health messages. We hope that you will.

A Special Case of Exaggerated Numbers: Survival Statistics

If there were a hall of fame for exaggeration, survival statistics would get a lifetime achievement award. Survival statistics are widely used. Unfortunately, they are also widely misused and easily misunderstood, even when you know the starting and modified risks. Because these statistics come up so often in cancer screening, we’d like to explain how they work.

Cancer screening means testing people who have no symptoms to look for hidden, early evidence of cancer. The assumption is that if you find cancer in very early stages, when a tumor is small, it is less likely to have spread to other parts of the body and is more likely to be curable.

There are many tests that can be used for screening, including mammography for breast cancer, colonoscopy for colon cancer, and PSA testing for prostate cancer. You should be clear that how a test gets used determines whether it is a screening test. When a woman who has no signs or symptoms of breast cancer goes for her annual mammogram, she is getting a screening test. But when a woman feels a lump in her breast and gets a mammogram, she is not getting a screening test—she is getting a diagnostic test in response to a symptom.

You’ll often hear survival statistics quoted to demonstrate the value of screening. The most common statistic used is a 5-year survival rate. There’s nothing special about a span of 5 years. Survival statistics can be calculated for any time period. The lung and prostate cancer examples we’ll discuss in the next pages use both 5- and 10-year survival statistics. For either time frame, the issues that you need to understand are identical.

Here’s a typical example of how survival statistics are misused. This example is an excerpt from a story published in the Los Angeles Times in 2006:1

Study Calls for Routine CT Scans for Smokers

The use of advanced CT imaging to detect Lung tumors in their still-treatable early stages greatly increases survival rates, and smokers should be routinely screened just as women are for breast cancer, according to a report today in the New England Journal of Medicine.

Imaging yielded an estimated 10-year survival rate of more than 90%, researchers said. Currently, about 5% of the 174,000 lung cancer patients diagnosed each year survive for 10 year. . . .

“This is compelling evidence that you can use CT screening to find lung cancer . . . and when you find it early and take it out early, you can cure a high percentage of patients,” said Dr. Claudia I. Henschke of Cornell University’s Weill Medical College in New York City, who led the study.

These results sound almost miraculous: CT scans for smokers seem to change the 10-year survival rate for lung cancer patients from 5 percent to over 90 percent. Does this prove that CT screening works? The short answer is no. To understand why not, you have to consider exactly how a 10-year survival rate is calculated. It is a fraction that, in this case, tells you what proportion of a group of people diagnosed with lung cancer is still alive 10 years later. Imagine 1,000 people diagnosed with lung cancer 10 years ago. If 50 are alive today, the 10-year survival rate is 50/1,000, or 5 percent. If 900 are alive today, the 10-year survival rate is 900/1,000, or 90 percent. Yet even if CT screening raised the 10-year survival rate to 90 percent, it is entirely possible that none of these patients will get an extra day of life.

The best way to understand this is to work through two thought experiments. First, consider a group of people with lung cancer who will all die at age 70. If they first receive the diagnosis when they are 67, their 10-year survival rate is 0 percent. (Do you see why? If they all die at age 70, none will be alive 10 years later.) But if these same people had received CT scans that found their cancer before they had symptoms, they would have been diagnosed earlier—at, say, age 57—and their 10-year survival rate would have been 100 percent. Yet death would still come at age 70. Earlier diagnosis always increases survival rates, but it doesn’t necessarily mean that death is postponed. This diagram helps you review how this happens.

Image ucalpkycf14.jpg

In other words, making the diagnosis earlier does not in itself mean that you delay death. It may simply mean that you know about the cancer for a longer time.

Comparing survival rates when people are diagnosed with a disease in different ways—for instance, based on a CT scan versus based on symptoms—is always misleading. That’s because the CT scans are great at finding early, hidden cancers, and survival rates always go up when diagnoses are made in early stages.

A second thought experiment helps to explain why CT scans, which find so many minute tumors, can inflate survival rates even if no lives are saved. It turns out that some forms of cancer look like cancer under the microscope but do not behave the way we might expect cancer to—that is, they do not develop into a relentlessly progressive, deadly disease. These nonprogressive cancers grow so slowly that they never cause symptoms, nor do they affect how long a person lives. Consequently, the only way we can diagnose these nonprogressive tumors is by screening.

Prostate cancer provides the most familiar example. We can find microscopic evidence of prostate cancer in around half of 60-year-old men if we look hard enough. Yet only 3 in 1,000 will die from prostate cancer over the next 10 years. How can this be? Because prostate cancer isn’t just one disease: it’s a spectrum of disorders. Some forms of prostate cancer grow very rapidly and kill the men who have them. Some grow so slowly that, even without treatment, men die of something else before the cancer causes symptoms. And other forms look like cancer under the microscope but never grow at all or may regress spontaneously. While this phenomenon is best understood in prostate cancer, it probably occurs in all cancers.

Now imagine a city in which 1,000 people are found to have lung cancer following evaluation for cough and weight loss (symptoms of lung cancer). At 10 years after diagnosis, 50 are alive and 950 have died, which equals a 10-year survival rate of 5 percent. Now let’s start again—but this time imagine that everyone in the city is screened with CT scans. In this scenario, 5,000 are given a cancer diagnosis: the 1,000 who had symptoms, plus 4,000 others who actually have nonprogressive cancers. These 4,000 would not die from lung cancer in 10 years. Consequently, the 10-year survival rate for lung cancer in the city would increase dramatically—to 81 percent—because these healthy people would appear in both parts of the fraction: 4,050/5,000. But what has really changed? Some people were unnecessarily told that they had cancer (and may have undergone harmful therapy), but the same number of people (950) still died. We have sketched out this scenario in the following diagram.

Image ucalpkycf15.jpg

The idea that screening can increase survival rates without actually saving lives is not just a theory. It was confirmed in a randomized trial of regular chest X-ray screening at the Mayo Clinic involving more than 9,000 male smokers.2 In that trial, 5-year survival rates were almost two times higher for those who had been screened than for those who had not been screened (35 percent versus 19 percent), but death rates were the same in the two groups. In fact, the death rate was slightly higher in the screened group. Consequently, doctors do not recommend lung cancer screening with chest X-rays.

Survival statistics and death statistics sound like two ways of talking about the same thing. But they are not flip sides of the same coin. To understand how this works, it helps to look at how these kinds of statistics are calculated.

Let’s start with survival statistics. As an example, let’s calculate the 5-year survival rate for a particular type of cancer (we’ll call it cancer X). As explained earlier, the 5-year survival rate for a group of patients diagnosed with cancer X is the number of people in the group who are alive 5 years after their cancer diagnosis, divided by the number originally diagnosed. Here’s the formula:

5-year survival rate=number of patients alive 5 years after diagnosisnumber of patients originally diagnosed with cancer X
Now let’s calculate a death statistic, which typically refers to a 1-year time frame (you can think of it as presenting a 1-year risk of death). For example, the death rate (or mortality rate, as researchers call it) for cancer X is the number of people in a group who die from that cancer over 1 year, divided by the number of people in the group. Here’s the formula:
Annual death rate=1-year risk of death=number of people who died from cancer X over 1 yearall people in the group
The key thing to notice is how the denominator (the bottom number in the fraction) for survival statistics differs from the denominator for death statistics. For survival rates, the denominator is the number of patients diagnosed with the illness; for death rates, it is all the people in a group (which could be, for instance, the population of a country, an age group, or some other defined group).

Survival statistics are always distorted by screening. As you saw in the last two illustrations (pages 91 and 93), survival statistics can go up even when the same number of people die. So survival statistics cannot tell you whether fewer people are dying from cancer.

In contrast, death rates can tell you what is really happening. If finding early cancer delays death, the annual death rate indeed goes down, since fewer people are dying in that year. But if screening is finding only more nonprogressive cancer, the death rate doesn’t change.

But comparing survival rates and death rates is like comparing apples and oranges. In fact, when you look at our best national cancer data, you will find no relationship between improved 5-year survival rates and death rates over time.3 For example, the 5-year survival rate for melanoma, the most deadly form of skin cancer, has improved from 49 percent in 1950 to 92 percent in the most recent data available. But death rates have actually gone up during this same period, from 1 death in 100,000 to almost 3 deaths in 100,000.4

So when you hear about changes in survival rates among people who have been screened for a disease, you cannot automatically assume that these changes reflect fewer deaths from the disease. Such changes could simply mean that we are diagnosing more early cases of disease but not delaying death, or that we are finding more nonprogressive disease. In other words, improved survival rates among people who have been screened for a disease provide no evidence that screening works to reduce deaths. The only way to know whether screening works is to show that death rates are lower in the screened group. Screening may well reduce deaths from a disease, but survival statistics are never a good guide to whether this is happening.

When used for the right purpose, however, survival statistics can be a good guide. The Learn More box on the next page shows you how survival statistics can be used appropriately to estimate your prognosis (your chance of surviving for a fixed period of time with a particular disease) and to judge the benefit of a treatment.

Box Icon

Box

Learn More. Although survival statistics are misleading in assessing the value of screening, they are not misleading when used for two other purposes: as a prognosis for an individual patient and as an outcome in a randomized trial. When you are diagnosed (more...)

This discussion about survival rates and death rates may seem a little counterintuitive. To help make sure you understand the ideas we’ve covered, take a look at the following chart, which appeared in a 1990 brochure sent to patients by the M. D. Anderson Cancer Center, a major academic medical center in Texas.

Image ucalpkycf16.jpg

QUIZ

Do prostate cancer patients treated at M. D. Anderson Cancer Center live longer than the average U.S. patient?

  1. Yes
  2. No
  3. Can't tell

EXTRA CREDIT: Out of 1,000 people diagnosed with cancer X, 400 are alive 5 years later. This means that cancer X has a 5-year survival rate of 40 percent.

If another 1,000 people are diagnosed with nonprogressive cancer, what will the 5-year survival rate be for the total group of 2,000?

HINT: Remember that people with nonprogressive cancer should be included in both the numerator and the denominator of the fraction.

  1. 10 percent
  2. 40 percent
  3. 70 percent

The correct answer to the first question is c. It turns out that U.S. survival rates for prostate cancer—not just the survival rates for M. D. Anderson’s patients—also went up dramatically during this period, from about 40 percent in I960 to 90 percent in 1990. This increase in 5-year survival reflects the growth of prostate cancer screening with the PSA blood test. In fact, the average U.S. 5-year survival rate of 90 percent was, ironically, higher than the survival rate at M. D. Anderson. To judge whether fewer people are dying of prostate cancer, death rates are the right measure to use. Unfortunately, this line in the chart looks pretty flat—suggesting that deaths from prostate cancer in the United States did not change much over this time. Comparing the 5-year survival rates of M. D. Anderson’s patients with overall U.S. mortality rates is nonsense.

Learn More

Understanding the Benefits and Downsides of Screening

Understanding screening, survival statistics, and why an improved survival rate does not equal fewer deaths is tricky. Many people (even physicians) get this wrong. If you are interested in reading more about this topic, we recommend Dr. H. Gilbert Welch’s book Should I Be Tested for Cancer? Maybe Not and Here’s Why (Berkeley: University of California Press, 2004).6

The correct answer to the Extra Credit question is c, 70 percent. Here’s how we did the calculations. First, we wrote out the fraction for the 5-year survival rate of 40 percent:

400 patients with cancer X alive 5 years after diagnosis1,000 patients originally diagnosed with cancer X=40%
Then we added the 1,000 people with nonprogressive cancer to both the numerator and the denominator:
400 patients with cancer X + 1,000 patients with nonprogressive cancer X alive 5 years after diagnosis1,000 patients diagnosed with cancer X + 1,000 diagnosed with nonprogressive cancer X=1,4002,000=70%
Although this example may be a little difficult to follow at first, it allows you to see for yourself how dramatically 5-year survival rates can change when nonprogressive cancers are diagnosed—although no one has lived any longer.

Copyright © 2008, The Regents of the University of California.

Know Your Chances: Understanding Health Statistics is hereby licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported license, which permits copying, distribution, and transmission of the work, provided the original work is properly cited, not used for commercial purposes, nor is altered or transformed.

Bookshelf ID: NBK126168

Views

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...