NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Evans I, Thornton H, Chalmers I, et al. Testing Treatments: Better Research for Better Healthcare. 2nd edition. London: Pinter & Martin; 2011.

Cover of Testing Treatments

Testing Treatments: Better Research for Better Healthcare. 2nd edition.

Show details

Chapter 5Dealing with uncertainty about the effects of treatments

In this chapter we look at the uncertainties that almost invariably surround the claimed effects of treatments, whether new or old. For example, few would probably question the routine use of oxygen in people who have had a heart attack, yet there is no good evidence that it helps, and some evidence that it may cause harm. This uncertainty has never been adequately addressed1 and many other effects of treatments are disputed.


Only rarely will the evidence be so clear-cut that there is no room for doubt about whether a treatment works.2 In such cases the treatment effect is often dramatic and immediate. Take the heart rhythm disorder known as ventricular fibrillation, where muscle contraction in the ventricles (lower chambers) of the heart becomes wildly uncoordinated. This is a medical emergency – death can occur in minutes. The technique of ‘zapping’ the heart with a direct electrical current from a defibrillator applied to the chest is used to restore the heart’s normal rhythm; when successful, the effect is virtually instantaneous.

Other dramatic effects (see also Chapter 6, p70) include drainage of pus to relieve pain from abscesses, blood transfusion for shock caused by severe haemorrhage, and insulin (a hormone produced by the pancreas) for diabetes. Up to the 1920s, patients with diabetes had short lives and suffered immensely, wasting away with uncontrollably high blood sugar levels. Very quickly, the initial results of animal tests led to the use of insulin in patients, with outstanding success – their response was near miraculous at the time. Another example from that era was the use of liver – later shown to be a source of vitamin B12 – for patients with pernicious anaemia. In this then fatal type of anaemia, the numbers of red blood cells gradually fall to disastrously low levels, leaving patients with a ghostly pallor and profound weakness. When these patients were given liver extract they recovered rapidly, and vitamin B12 is now prescribed routinely for this form of anaemia.

Some examples from the beginning of this century highlight similarly dramatic results.

Laser treatment of portwine stains

The birthmarks known as portwine stains are caused by permanent and malformed dilated blood vessels in the skin. Commonly occurring on the face, they persist and often darken as the child matures, and can be seriously disfiguring. Numerous treatments were tried over the years including freezing, surgery, and radiation, but with little impact and many side-effects. The introduction of laser treatment brought impressive results: improvement is usually seen after a single laser session in most types of lesions, and the damage caused by dispersion of heat from the laser to the surrounding skin tissues is temporary.2, 3

Imatinib for chronic myeloid leukaemia

Impressive results have also been seen in patients given imatinib for chronic myeloid leukaemia.4, 5

Before imatinib was introduced in the late 1990s, this type of leukaemia responded very poorly to standard treatments. When the new drug was tried, initially in patients who had not responded to standard therapy, the outlook for patients improved greatly. Imatinib stabilizes the disease, appears to prolong life substantially by comparison with the pre-imatinib era, and has mostly mild side-effects. It is now regarded as the first treatment option.

Mother’s kiss

Low-tech approaches can have dramatic effects too. Young children sometimes place small objects – plastic toys or beads, for example – in their nose. But they often have trouble blowing their nose to expel such foreign bodies. The ‘mother’s kiss’ technique for dislodging the offending object – involving a parent closing the unblocked nostril while blowing into the child’s mouth – is simplicity itself, as well as being very effective.2, 6

A new treatment for strawberry birthmarks

Treatments with dramatic effects are occasionally discovered by accident. Take the example of a condition that occurs in infants called a haemangioma, which, like portwine stains, is also due to malformation of immature blood vessels. In haemangiomas, small blood vessels come together to form a lump. Haemangiomas mostly affect the skin, usually on the head and neck, but they can occur in organs inside the body such as the liver. The skin lesions, which are often called strawberry marks because of their bright red, raised appearance, are not usually visible at birth but generally appear in the first week or so of life. They tend to grow rapidly in the first three months and then the growth rate slows. In most cases they disappear of their own accord by the time the child is five years old, leaving behind a faint pink mark or some loose skin.

However, some haemangiomas need treatment because of their position – for example, they may cover an eye or block the nose. Or treatment may be necessary because of other complications. Ulcerated haemangiomas may become infected, or heart failure may develop in patients with very large lesions because the heart has to pump so much blood through blood vessels in the lump.

Until recently, steroids were the first-choice medical treatment for problematic haemangiomas. Then in 2008, some doctors had dramatic results with another treatment, which they came across quite by chance. They were using steroids to treat a baby with a huge haemangioma that almost swallowed up the face and right eye. Despite this treatment, however, the baby developed heart failure. So, to treat the heart failure they started the baby on a standard drug for this condition called propranolol. To their astonishment, the appearance of the haemangioma started to improve within 24 hours, and within a week the tumour had shrunk sufficiently for the baby to open an eyelid. After six months of treatment the haemangioma had melted away. Over the following year the doctors went on to use propranolol in a dozen children with similar success. These impressive results have been replicated by other doctors in small numbers of children and propranolol is now being studied further in larger numbers of infants.7,8


‘Science itself works very badly as a news story: it is by its very nature a subject for the “features” section, because it does not generally move ahead by sudden, epoch-making breakthroughs. It moves ahead by gradually emergent themes and theories, supported by a raft of evidence from a number of different disciplines on a number of different explanatory levels. Yet the media remain obsessed with “new breakthroughs”.’

Goldacre B. Bad Science. London: Fourth Estate, 2008, p219.


Most treatments do not have dramatic effects and fair tests are needed to assess them. And sometimes a treatment may have a dramatic effect in some circumstances but not in others.

Although vitamin B12 is undoubtedly effective for pernicious anaemia (see above), dispute continues to this day about whether patients need quarterly or more frequent treatment. That question will only be answered by carefully controlled tests comparing the options. Moreover, whereas the pain relief with hip replacements is dramatic, the relative merits of different types of artificial hip joints are far more subtle, but may nevertheless be important – some may wear out faster than others for example. With laser treatment of portwine birthmarks (see above), there is also still much to learn. Whilst this treatment remains the ‘gold standard’, research continues into why some lesions re-darken after several years, and on the effects of different types of lasers, possibly combined with cooling of the skin.9, 10

And while aspirin substantially reduces the risk of death in patients suffering a heart attack if given promptly on diagnosis, whether taking aspirin to prevent heart attacks and strokes does more harm than good depends on whether patients have underlying cardiovascular disease. The benefits – reduction in the risk of heart attacks, strokes, and death from cardiovascular causes – need to be balanced against the risks – bleeding, especially the type of stroke caused by bleeding into the brain, and bleeding from the gut. In patients who already have cardiovascular disease, the benefits of the drug greatly outweigh the risks. But in otherwise healthy people, the benefits of aspirin do not clearly outweigh the risk of bleeding (see Chapter 7).11


For many diseases and conditions, there is substantial uncertainty about the extent to which treatments work, or about which treatment is best for which patient. That doesn’t stop some doctors having very strong opinions about treatments, even though those opinions may differ from one doctor to the next. This can lead to considerable variation in the treatments prescribed for a given condition.

In the 1990s, Iain Chalmers, one of the authors, while holidaying in the USA, broke an ankle and was treated by an orthopaedic surgeon. The surgeon put the leg in a temporary splint, and said that the next step, once the swelling had subsided, would be a lower leg plaster cast for six weeks. On returning home a couple of days later, Iain went to the local fracture clinic, where a British orthopaedic surgeon, without hesitation, dismissed this advice. Putting the leg in plaster, the British surgeon said, would be wholly inappropriate. In the light of this obvious professional uncertainty, Iain asked whether he could participate in a controlled comparison to find out which treatment was better. The British surgeon answered that controlled trials are for people who are uncertain whether or not they are right – and that he was certain that he was right.

How can such a pronounced difference in professional opinion come about, and what is a patient to make of this? Each surgeon was certain, individually, about the correct course of action. Yet their widely divergent views clearly revealed uncertainty within the profession as a whole about the best way to treat a common fracture. Was there good evidence about which treatment was better? If so, was one or neither surgeon aware of the evidence? Or was it that nobody knew which treatment was better (see Figure).

What should a doctor do?

What should a doctor do?

Perhaps the two surgeons differed in the value they placed on particular outcomes of treatments: the American surgeon may have been more concerned about relief of pain – hence the recommendation of a plaster cast – while his British counterpart may have been more worried about the possibility of muscle wasting, which occurs when a limb is immobilized in this way. If so, why did neither surgeon ask Iain which outcome mattered more to him, the patient? Two decades later, uncertainty continues about how to manage this very common condition.12

There are several separate issues here. First, was there any reliable evidence comparing the two very different approaches being recommended? If so, did the evidence show their relative effects on outcomes (reduced pain, or reduced muscle wasting, for example) that might matter to Iain or to other patients, who might have different preferences to his? But what if there was no evidence providing the information needed?


Failure to face up to uncertainties about the effects of treatments can result in avoidable suffering and death on a massive scale. If when diazepam and phenytoin were introduced as anticonvulsants for eclampsia they had been compared with magnesium sulphate – which had been in use for decades – hundreds of thousands fewer women would have suffered and died. Similarly, if the effects of systemic steroids for traumatic brain injury had been assessed before this treatment became widely adopted, tens of thousands of unnecessary deaths could have been avoided. These are just two examples of many that might have been used to illustrate why doctors have a professional responsibility to help address uncertainties about the effects of treatments.’

Chalmers I. Addressing uncertainties about the effects of treatments offered to NHS patients: whose responsibility? Journal of the Royal Society of Medicine 2007; 100: 440.

Some clinicians are clear about what to do when there is no reliable evidence about the effects of alternative treatments and are prepared to discuss this uncertainty with patients. For example, one doctor who specializes in caring for people with stroke, commented that, although research evidence shows that his patients would fare better if cared for in a stroke unit, it remained uncertain – for many types of patients – whether they should receive clot-busting drugs (see also Chapter 11, p139). When discussing treatment options with his patients he explained that these drugs may do more good than harm, but they may – for some patients – actually do more harm than good. He then went on to explain why, talking to a patient for whom the balance of risk and benefit was unclear, he felt he could only recommend this treatment for them within the context of a carefully controlled comparison, which should help to reduce the uncertainty.13 Uncertainties about several aspects of clot-busting drugs persist.14


‘One of the key attributes of professionalism . . . should be the ability to identify and address uncertainty in medicine. Every day professionals confront and cope with uncertainties about disease pathogenesis, about diagnosis, and about treatment. Yet the intrinsic uncertainties in all these spheres of medical activity are seldom acknowledged explicitly and some professionals remain uncomfortable about admissions of uncertainty – in their dealings with patients especially. Uncertainty is also a prime stimulus for medical research to improve human health, which is central to the MRC’s mission. In the future it will be increasingly important for medical professionals to take on board the results of accumulated research findings relevant to their area of practice so that they are aware where continuing uncertainties exist and what research is ongoing or needed to address these. Overall, a mark of professionalism for the future will be research awareness for the benefit of patients. Some medical professionals will actively participate in research but all should seek to encourage it and, where appropriate, to involve their patients actively in the medical research agenda, and implement the results of this research in their professional practice.’

From: Medical Research Council response to Royal College of Physicians consultation on medical professionalism. 2005

Caffeine for breathing problems in premature babies

Large variations in the treatments used for a particular condition provide clear evidence of professional uncertainty about the relative merits of different treatments. And entrenched practices may mean that it takes a very long time for such uncertainties to be addressed by fair tests. The use of caffeine in premature babies provides a telling example. Such babies often have trouble breathing properly and sometimes stop breathing very briefly – this condition is known as apnoea of prematurity and affects most babies born at less than 34 weeks’ gestation. In the late 1970s, caffeine treatment was shown to reduce these episodes and then became used by some paediatricians.

However the effects of caffeine remained disputed. Although fair tests had shown that caffeine reduced the episodes of apnoea, many paediatricians did not think that the episodes were sufficiently serious to justify use of the drug, and some were concerned that it might not be safe in these tiny babies. This meant that some babies were given the treatment and others weren’t. When these widespread uncertainties were finally assessed by a large international study more than 30 years after the treatment had been introduced, it turned out that this simple therapy not only reduces the breathing difficulties but also, and very importantly, significantly improves the likelihood of long-term survival without cerebral palsy and delay in infant development. Had this uncertainty been addressed when the treatment was introduced, fewer babies would have gone on to develop disabilities.15, 16

Antibiotics in pre-term labour

Fair tests of treatments with hoped-for beneficial effects, and which are assumed to be harmless, can show that neither is true. Doctors prescribe treatments with the best of intentions, particularly when they may offer hope in a desperate situation. For example, a theory suggested that ‘silent’ (sub-clinical) infection might trigger early labour and preterm delivery. The theory led doctors to prescribe antibiotics for some pregnant women in the hope that this might help to prolong pregnancy. No one seriously thought that using antibiotics in this way would cause any serious problems. Indeed, there is some evidence that women themselves were keen to have antibiotics – in a spirit of ‘let’s try this; it can’t do any harm’.

When a fair test of this treatment was eventually done, the results had clear clinical implications. For a start, no benefits were identified. On top of that, long-term follow-up of the babies in the study showed that those who had been exposed to antibiotics were more likely than those in the comparison groups to have cerebral palsy and problems with speech, vision, and walking. These risks of antibiotics had remained unrecognized over the decades that antibiotics had been prescribed to women, but without adequate evidence from fair tests about their effects. As often happens, those who were given an inadequately evaluated treatment in ‘normal’ clinical practice were more likely to be harmed than those given the same treatment prescribed in a research context. Put another way, people were generally more at risk when they were not taking the drugs as part of a fair test.17, 18, 19


In a fictional conversation between two doctors, a general practitioner makes the following point:

‘Tons of what we do is guesswork and I don’t think that you or I feel too comfortable with that. The only way to find out if something works is a proper trial, but the hoops are huge. So what do we do? We do what we fancy. And I’m sure some of the time it’s fine – clinical experience and all that. Maybe the rest of the time we’re just as likely to be getting it wrong as right, but because whatever we’re doing isn’t called a trial, no one regulates it and none of us learn from it’.

Adapted from Petit-Zeman S. Doctor, what’s wrong? Making the NHS human again. London: Routledge, 2005, pp79–80.

Breast cancer

The treatment of breast cancer (see Chapter 3) provides another example of professional uncertainty. There is considerable variability in the use of surgery, radiotherapy, and chemotherapy. The best treatment of very early stage breast cancers and of ‘pseudo-cancers’ of the breast is unresolved, as is the ideal number of lymph nodes to remove from the armpit, or indeed whether any should be removed at all.20 As if that were not enough, topics of particular interest to patients, such as relief of fatigue associated with therapy, or the best way of treating lymphoedema of the arm – a distressing and disabling aftermath of surgery and radiotherapy in the armpit – still have not been tested adequately.


Where do we go from here? Clinicians need to be able to draw on resources that provide the best current evidence about a treatment, taken from collective experience and systematic reviews of any reliable research studies that exist. If, after doing this, they find that uncertainty remains about a treatment, they need to be prepared to discuss this with their patients and to explain why this is so. Patients and clinicians can then discuss the options together, taking into account patient preferences. These discussions may raise further uncertainties that need to be acknowledged and addressed. Only by recognizing together that uncertainties still exist, can steady progress be made towards making treatments more appropriate and safer. Uncertainty is therefore a prerequisite for progress, not an admission of ‘defeat’. This positive attitude to addressing uncertainties is now reflected in some professional guidance. In the UK, the General Medical Council’s latest version of its Good Medical Practice guidance instructs doctors that, as part of maintaining and improving their performance, they ‘must help to resolve uncertainties about the effects of treatments’.21 To do this, patients and clinicians must work together to design better research (see Chapter 11).


So what should happen when there is important uncertainty about the effects of new or old treatments that have not been properly evaluated? An obvious answer is to follow the example of the doctor caring for his stroke patients, as we described above: address the uncertainty by offering inadequately assessed treatments only within the context of research that has been designed to find out more about both their wanted and unwanted effects.

A medical ethicist put it this way:

‘If we are uncertain about the relative intrinsic merits of any [different] treatments, then we cannot be certain about those merits in any given use of one of them – as in treating an individual patient. So it seems irrational and unethical to insist one way or another before completion of a suitable trial. Thus the answer to the question, “What is the best treatment for the patient?” is: “The trial”. The trial is the treatment. Is this experimentation? Yes. But all we mean by that is choice under uncertainty, plus data collection. Does it matter that the choice is “random”? Logically, no. After all, what better mechanism is there for choice under uncertainty?’22

Providing treatments as part of fair tests can help to make a profound difference to outcomes for patients. The story of childhood leukaemia provides a very dramatic example of this. Until the 1960s, virtually every child with leukaemia died soon after the diagnosis had been made. Now about 85 children out of 100 survive. This has been achieved because most children with leukaemia have participated in randomized trials comparing the current standard treatment with a new variant of that treatment.23 For most children with cancer, therefore, the best treatment option is chosen by participation in such trials.


‘So where are we with addressing uncertainties about the effects of treatments? . . . Despite general acknowledgement that patients are partners in medical research and healthcare decisions, the complexity of discussing therapeutic uncertainty is unnerving some doctors. Some are simply fearful of provoking anxiety – doubtless a genuine concern but nevertheless paternalistic. Others try to justify their actions in terms of a balance between two ethical arguments – whether the ethical duty to tell the truth extends to being explicit about uncertainties versus the moral obligation to protect patients from emotional burden. Are patients prepared to live with uncertainty? We need to find out. Perhaps people are far more resilient than doctors suspect.’

Evans I. More nearly certain. Journal of the Royal Society of Medicine 2005;98:195–6.

If no such trial is available, at the very least the results of using new and untested treatments should be recorded in a standardized way – for example, by using a checklist of items including the laboratory or other tests that will be used to diagnose a condition and the tests that will be done to assess the impact of the treatment. The plan of investigation could also be registered in a database, as should happen for clinical trials (see Chapter 8). By doing this, the results can contribute to the body of knowledge for the benefit of the patients receiving the untested treatment and patients everywhere. Huge sums of money have already been invested in healthcare IT systems, which could readily be used to capture this information for the benefit of patients and of the public (see also Chapter 11).24

There will have to be changes if uncertainties about the effects of treatments are to be addressed more effectively and efficiently. We discuss some of these – particularly the greater involvement of patients – later in the book (see Chapters 11 and 12). However, there is a particular issue – we touched on it above – that we want to emphasize here. When there is insufficient information about the effects of a treatment, knowledge can be increased by ensuring that clinicians only offer it within the context of a formal evaluation until more is known about its value and possible disadvantages. Yet some prevailing attitudes, including systems of research regulation (see Chapter 9), actually discourage this risk-limiting approach.

The problem vexed a British paediatrician over 30 years ago when he pithily observed that he needed permission to give a treatment to half his patients (that is, to find out about its effects by giving half the patients the new treatment and half the existing treatment in a controlled comparison), but not if he wanted to give the same treatment to all of them as a standard prescription.25 This illogical double standard still pops up repeatedly and discourages clinicians who want to reduce uncertainties about the effects of their treatments. The overall effect is that health professionals can be deterred from generating knowledge from their experiences in treating patients. As the American sociologist Charles Bosk once remarked: ‘anything goes, as long as we promise not to learn from the experience’.

Being able to explain uncertainty clearly demands skills and a certain degree of humility on the part of doctors. Many feel uneasy when trying to explain to potential participants in a clinical trial that no one knows which treatment is best. But the public’s attitude has changed: arrogant doctors who ‘play God’ are increasingly given short shrift. We need to focus on training doctors who are not ashamed to admit they are human and that they need the help and the participation of patients in research to provide more certainty about choices of treatments (see Chapters 11 and 12).

The main stumbling block for many clinicians and patients is lack of familiarity with the features of fair tests of treatments, an issue we tackle next (see Chapter 6).


  • Dramatic effects of treatments are rare
  • Uncertainties about the effects of treatments are very common
  • Small differences in the effects of different treatments are usual, and it is important to detect these reliably
  • When nobody knows the answer to an important uncertainty about the effects of a treatment, steps need to be taken to reduce the uncertainty
  • Much more could be done to help patients contribute to reducing uncertainties about the effects of treatments
Copyright © 2011 Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou.
Bookshelf ID: NBK66195


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...