The weakness of one

Les Irwig; Judy Irwig; Lyndal Trevena; Melissa Sweet

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Irwig L, Irwig J, Trevena L, et al. Smart Health Choices: Making Sense of Health Advice. London: Hammersmith Press; 2008.

Chapter 8The weakness of one

Theists, for example, note the number of times their prayers have been answered and conclude that there is a benevolent god; atheists cite the occasions that their prayers have gone unanswered and conclude that we are on our own. Both need to develop the habit of thinking more broadly. Both must consider the number of times their hopes have been answered when they have prayed and when they have not, as well as the number of times their hopes have been dashed when they have prayed and when they have not
Thomas Gilovich¹

One of the points raised in Chapter 7 is the compelling allure of personal testimony. For many of us, this is one of the most seductive sources of health information. A neighbour says her cancer disappeared after she took shark cartilage. Your mother swears that taking a vitamin C tablet every morning keeps her free of colds. A colleague claims his bad back recovered after doing a certain exercise for 6 weeks.

It can be tempting to draw conclusions from such anecdotes; somehow a story involving a real person whom you know can seem more convincing than the results of studies based on thousands of anonymous participants. Anecdotal evidence is usually based on individual experiences or observations, as distinct from probabilistic evidence that gives estimates of how likely something is to occur based on experience with large numbers of people. In this chapter we discuss some of the ways in which stories can be helpful in making health decisions, but also warn of their limitations.

The danger of the anecdote

There are inherent dangers in relying totally on anecdotes. Consider the case of Mr Dickens, 70, who recently consulted Dr Carter about an irregular pulse. Mr Dickens, who has previously had high blood pressure and a stroke, is found to have a disturbance of his heart rhythm, called atrial fibrillation. This condition may cause a blood clot to develop in the heart and send off fragments that can cause a stroke by blocking arteries in the brain. One treatment used in people with atrial fibrillation is an anticoagulant, which prevents a blood clot. But Dr Carter knows that this drug can also cause internal bleeding, with potentially serious consequences, although she has never had a patient suffer this particular side effect. After she prescribes the treatment, Mr Dickens has a bleed into the brain.

Soon after, Dr Carter sees Mr Jones, another elderly man with similar problems to Mr Dickens. But Dr Carter does not prescribe an anticoagulant this time, discouraged by her recent experience with Mr Dickens. Mr Jones later suffers a stroke. Dr Carter will never know if this might have been prevented if she had prescribed an anticoagulant. But an examination of the probabilistic evidence – as distinct from the anecdotal evidence provided by case reports – gives us some idea.

If Dr Carter had done a literature search, she would have found several good randomised controlled trials showing a two-thirds reduction in stroke for patients treated with anticoagulants. On the other hand, serious bleeds from anticoagulation are rare, so overall her patients with atrial fibrillation would be served best by taking anticoagulants unless they are at low risk of stroke or at high risk of bleeding. Mr Dickens is at high risk of stroke because he has high blood pressure and has had a previous stroke. Out of 1000 people like Mr Dickens who are treated, about 120 strokes would be prevented in the next year, whereas a bleed into the brain as a result of anticoagulants would occur in about 5 people. In addition, there may be other bleeding, some of which would be mainly a nuisance such as bruises, and some of which could be more serious, such as bleeds into the stomach or bowel.

It [an anecdote] is useful for documenting that the outcome can occur, but provides no information about the frequency with which it occurs or the effect of an intervention on the frequency of occurrence.
David Eddy²

Only survivors speak!

Anecdotes have limited use in judging the effectiveness of health interventions. If you wanted to know, for example, whether a certain cancer treatment saves lives, the opinion of someone who had the treatment would not be a reliable guide. Remember that those patients in whom the treatment did not work are no longer around to give their views. Only survivors speak – which can result in a very biased picture of an intervention.

But anecdotes can be useful in some situations

It is generally unwise to rely on other people’s stories as a guide to how likely you are to experience similar benefits or harms from an intervention. However, anecdotes are useful in some situations.

When confronted by illness or other health problems, many people find it helpful to talk to others who have been through similar situations. Their stories can provide useful insights into how your life might be affected by a similar illness, or side effects from a treatment, and what strategies might be useful in helping you deal with them. Indeed, some universities now invite patients to talk to medical students about their own experiences with illness, in an attempt to ensure that doctors become more understanding of and sympathetic to what it is like to be a patient. Other people’s stories can also provide useful information on how to find your way around the health system, which can seem like a confusing maze to a newcomer. Patient support groups can be particularly useful in these situations.

Most scientific and medical discoveries have their roots in anecdotes, which have led to hypotheses that are then proved by rigorous testing. In some circumstances, the anecdotal evidence can be so spectacularly convincing that the need for further confirmation diminishes. For example, when Howard Florey and Ernst Chain developed the drug penicillin based on Alexander Fleming’s earlier work, the antibiotic properties were so striking that it was introduced for use without long-term trials. When people are treated for an illness and survive in the face of evidence that most people die without treatment, there is usually little doubt about the treatment’s efficacy.

Let’s explore a situation where personal experience may help to decide whether a treatment works. Consider the case of Ed Smith, who suffers from severe migraines. When a migraine strikes, Ed is incapacitated and has to lie down for several hours. He has tried many supposed remedies over the years, but none has worked. He hears from a friend about a new therapy that has helped her. He tries it the next time he feels a migraine starting, and his pain disappears quickly.

Is Ed’s excitement about the treatment well founded? He knows that the pain disappeared quickly when he took the tablets, and that they helped his friend. He now also needs to know whether he will experience the same immediate benefit when he takes the tablets again. If indeed he tries it again and he experiences the same pain relief, he probably has good reason to feel excited.

As with the introduction of penicillin, anecdotal evidence can be used to assess the affect of a treatment if at least some of several principles are fulfilled:

The outcome of the disease or condition is predictable in the absence of the treatment. The condition in question does not usually get better on its own, at least not immediately.
The effect of the treatment is immediate. The outcome is evident soon after the treatment.
The effect of the treatment is large. There is a dramatic, large and obvious effect that would be difficult to attribute to spontaneous improvement.
The effect of the treatment can be confirmed by repetition. If the nature of the condition is such that it recurs, it is possible to confirm the treatment’s effects by repeated testing.

Acting on someone else’s anecdotal experience is appropriate only if the harm seems small and the benefit worthwhile. Suppose that you suffer from chapped, itchy skin in winter, and a friend tells you about a new cream that helped him. Should you try it? It sounds like you should. First, if it relieves the dryness and itch, you can be fairly sure the cream is responsible if previous treatments have failed to make any difference. Second, because treatment is likely to be short term, the risks of serious adverse effects are low. Third, judging whether the cream is effective is straightforward. So, if the cream works, you will benefit and, if it doesn’t, you stand to lose very little.

N of 1 trials

We could be more scientific about assessing the effect of the anti-itch cream, even with just one person, by using what we call an ‘N of 1’ trial. These trials have been defined as:

... [a trial where] the patient undergoes pairs of treatment periods organised so that one period involves the use of experimental treatment and the other involves the use of an alternate or placebo therapy. The patient and physician are blinded, if possible, and outcomes are monitored. Treatment periods are replicated until the clinician and patient are convinced that the treatments are definitely different or definitely not different.

If we had a ‘fake’ or placebo cream as well as the active one and tried each, one at a time, without realising which one we had tried, this would be an ‘N of 1’ trial. Ed Smith could have done one of these for his migraine treatment if he did not have labels on the pills and tried the new one against a fake one without knowing which was which.

In short, anecdotal information is useful when you are looking for immediate symptomatic relief for a relatively minor condition, and there is little potential for the treatment to do harm. It is also useful if you want to know how other people coped with a specific problem, or gain some insight into their experiences of diseases or interventions. It might be helpful for generating hypotheses that can be more rigorously tested.

A broad range of stories

But what if we have a collection of stories on the same topic? This may help us to learn about the context of what can happen in an illness or the likely sequence of events.³ For example, the same strain of flu can render one person sick in bed for a fortnight whereas another manages to keep functioning and is better within 4 or 5 days. People’s stories can be used to develop concepts and hypotheses, which can then be more rigorously tested and evaluated with a large group of people and in different settings.

The usefulness of a broad range of stories depends on how they have been collected and analysed. On most topics, if you listen to enough people’s stories, eventually you will start to hear similar accounts of the illness or treatment in question. As a very rough guide, this usually happens when you start to collect more than about 30 stories and you can be reasonably confident that you’ve captured the most common and likely experience of an illness or its treatment. Ideally two different researchers should look at recordings of the stories and identify the common themes within them.

A good example of this is a website called DiPEx (Database of Individual Patients’ Experience) at www.dipex.org. This website has over 100 modules on different illnesses and patient experiences. Each module consists of a number of patients’ stories that typify over 40 or 50 stories that were recorded on each topic. This means that the main patient experiences are more likely to be covered.

A resource like this is more powerful and useful than just one anecdote when you are trying to make a decision, because it is a bit more balanced and provides a range of experiences from a number of people, not just one perspective or opinion. However, although it gives a range of experiences, it does not provide information on how commonly they occur.

Summary

As seductive as anecdotal reports can be, it is usually unwise to rely on generalisations based on one or two experiences. They do not tell us the most probable outcome, which is most useful for guiding decisions. Anecdotal evidence is useful to help you understand the nature of the symptoms of a disease and of the side effects of treatment. However, anecdotes are poor evidence of how likely that outcome is to occur, except in a few circumstances as shown below:

Table 8.1Comparison between reliable and unreliable use of anecdotal evidence

Anecdotal evidence is reliable	Anecdotal evidence is unreliable
When the outcomes of the disease or condition are predictable in the absence of treatment (e.g. migraines, chronic arthritic pain, premenstrual tension)	When the outcomes of the treatment are uncertain for the individual (e.g. breast cancer, diabetes)
When the effects of the treatment are large	When the effects of the treatment are small and subtle
When the effects of the treatment are immediate	When the effects of the treatment are delayed
When the effects of treatment can be confirmed by repetition	When the effects of the treatment cannot be confirmed by repetition
When the effects of treatment can be confirmed by an ‘N of 1’ trial	When the effects of treatment are disproved by an ‘N of 1’ trial

References

1.: Gilovich T. How We Know What Isn’t So: the fallibility of human reason in everyday life. New York: Free Press; 1991. p. 187.
2.: Eddy D. Assessing Health Practices and Designing Practice Policies. USA: American College of Physicians; 1992.
3.: Greenhalgh T. What Seems to be the Trouble? Stories in illness and healthcare. Oxford: Radcliffe; 2006.

All rights reserved. No part of this publication may be reproduced, stored in any retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the publishers and copyright holder or in the case of reprographic reproduction in accordance with the terms of licences issued by the appropriate Reprographic Rights Organisation.

Bookshelf ID: NBK63643

Contents

< Prev Next >