• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of bmjBMJ helping doctors make better decisionsSearch bmj.comLatest content
BMJ. Feb 26, 2000; 320(7234): 537–540.
PMCID: PMC27295

Systematic reviews and meta-analyses on treatment of asthma: critical evaluation

Alejandro R Jadad, professor,a Michael Moher, Royal College of General Practitioners research training fellow,b George P Browman, professor,a Lynda Booker, research assistant,a Christopher Sigouin, doctoral student,a Mario Fuentes, research assistant,c and Robert Stevens, research assistantc

Abstract

Objective

To evaluate the clinical, methodological, and reporting aspects of systematic reviews and meta-analyses on the treatment of asthma and to compare those published by the Cochrane Collaboration with those published in paper based journals.

Design

Analysis of studies identified from Medline, CINAHL, HealthSTAR, EMBASE, Cochrane Library, personal collections, and reference lists.

Studies

Articles describing a systematic review or a meta-analysis of the treatment of asthma that were published as a full report, in any language or format, in a peer reviewed journal or the Cochrane Library.

Main outcome measures

General characteristics of studies reviewed and methodological characteristics (sources of articles; language restrictions; format, design, and publication status of studies included; type of data synthesis; and methodological quality).

Results

50 systematic reviews and meta-analyses were included. More than half were published in the past two years. Twelve reviews were published in the Cochrane Library and 38 were published in 22 peer reviewed journals. Forced expiratory volume in one second was the most frequently used outcome, but few reviews evaluated the effect of treatment on costs or patient preferences. Forty reviews were judged to have serious or extensive flaws. All six reviews associated with industry were in this group. Seven of the 10 most rigorous reviews were published in the Cochrane Library.

Conclusions

Most reviews published in peer reviewed journals or funded by industry have serious methodological flaws that limit their value to guide decisions. Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals.

Introduction

One of the most important challenges for anyone treating patients with asthma is the need to cope with vast amounts of information about an ever increasing number of new devices and drugs. The abundance of information often makes it difficult to establish the most appropriate drug or regimen to use in a given clinical situation.1

During the past two decades, interest has risen in the use of systematic reviews as tools to help decision makers cope with information overload. These reviews, in their ideal form, include an explicit description of how they were conducted and incorporate strategies to minimise bias and maximise precision.2,3 These reviews can also include meta-analysis (the statistical combination of the results of several independent studies to produce a single estimate of the effect of a particular intervention or healthcare situation).4 However, most systematic reviews and meta-analyses published in peer reviewed journals have methodological deficiencies that may limit their validity.512

Recently, important initiatives have emerged to improve the methodological quality of systematic reviews and meta-analyses. Perhaps the most prominent is the Cochrane Collaboration, a large international group11,13 that seems to be producing reviews that are more rigorous and better reported than those published in peer reviewed journals.9,14 We evaluated the quality of published systematic reviews and meta-analyses on the treatment of asthma and compared the characteristics of Cochrane reviews with those in peer reviewed journals.

Methods

Inclusion criteria

To be included, a report had to be described as a systematic review or a meta-analysis of the treatment of asthma and to be published in full, in any language or format, in a journal or the Cochrane Library.

Literature search and study selection

We identified eligible studies by a search up to July 1998 in Medline (from 1966), CINAHL (from 1982), HealthSTAR (from 1975), and EMBASE (from 1984) using a refined search strategy (see BMJ's website for details).9 This was complemented by a search of the Cochrane Library (issue 3, 1998) with the term “asthma” and searches of personal collections and the reference lists of eligible studies.

Independently, two of us (LB and ARJ) reviewed the citations identified by the search strategy to judge their eligibility. The final decision on inclusion in the review was made by consensus.

Data extraction

Three trained reviewers (MF, LB, RS) extracted data independently using unmasked reports.15 The final set was agreed by consensus, and differences were resolved by a fourth reviewer (ARJ).

Data were extracted on the general characteristics of the studies (authors, sources of funding, and source, year, place, and language of publication); clinical issues (population, intervention, condition, and outcomes reviewed), methodological characteristics (language restrictions; number, format, design, and publication status of the studies included; data synthesis; heterogeneity testing; and methodological quality); and conclusions. Methodological quality was assessed with Oxman and Guyatt's index, a validated tool that scores reviews on a seven point scale.8,16 Reports were regarded as having serious or extensive flaws if they received a score of 1 to 3, and as having minimal or minor flaws if they received scores from 4 to 7.

Data analysis

Reviews were divided according to publication in the Cochrane Library or journals and to the presence or absence of association with industry. The Wilcoxon rank-sum test (two sided) was used to compare the number of authors, included studies, and sources and the overall quality scores. Fisher's exact test (two sided) was used to compare the responses to the first nine items in Oxman and Guyatt's index and also the proportion of reviews that described the primary outcomes, selection criteria, and heterogeneity testing and the number of reviews with language restrictions between the Cochrane Library and journals. Values of P<0.05 were regarded as significant.

Results

General characteristics

We identified 50 eligible reviews (see BMJ's website for full list). The first review identified was published in 1988. Twenty nine (58%) of the reviews were published in 1997 and 1998. Twelve reviews were published in the Cochrane Library, and the remaining 38 were published in 22 peer reviewed journals. Only one review appeared in both groups. None of the reviews published in journals represented updated versions of earlier work. The journal with the highest number of reviews was Annals of Pharmacotherapy (seven) followed by the Journal of Allergy and Clinical Immunology and Respiratory Medicine (four each). Table Table11 gives other characteristics of the reviews.

Table 1
Characteristics of 50 reviews of treatment of asthma

Methodological characteristics

The number of studies included in the reviews varied from 2 to 97. Most of the reviews (37) included randomised controlled trials, but three included non-randomised trials with contemporaneous controls and one included case series. In 14 reviews (including some of those mentioned in the previous sentence) the exact type of studies was either unclear or not reported. Seven of the 50 reviews included unpublished articles and nine included articles available only in abstract form. None of the reviews published in peer reviewed journals had been updated.

Methodological quality

Forty reviews obtained quality scores of 3 or less, which is compatible with serious or extensive flaws. All six reviews associated with industry were in this group (table (table2).2). None of these reviews described efforts to reduce selection bias; one described the search strategy, one described the methods used to assess the quality of the studies, and three reported the methods used to combine data across studies (table (table2).2). All but one17 of these studies had results and conclusions that favoured the interventions related to the companies sponsoring the reviews. The number of reviews associated with industry was too small to warrant meaningful statistical comparisons with other reviews.

Table 2
Methodological quality of systematic reviews and meta-analyses of the treatment of asthma. Values are number of affirmative answers/number of reviews relevant to each question

Analysis of individual components of Oxman and Guyatt's index showed that 66% of the reviews stated the search methods used to find the evidence, 52% included a reasonably comprehensive search for evidence, 56% reported the inclusion criteria, 30% included measures to avoid selection bias, 28% reported the criteria used to assess the validity of the included studies, 24% used appropriate criteria for validity assessment, 52% reported the methods used to synthesise findings, and 50% had conclusions supported by the data or the analysis reported in the review (table (table2).2). In addition, 20 reviews (40%) included heterogeneity testing, 16 (32%) stated the primary outcome of interest to the reviewers, and eight (16%) included assessment of publication bias. We found no significant difference in the proportion of reviews with favourable results between those with minimal or minor flaws and those with serious or extensive flaws (1/10 v 11/40, P=0.416).

Cochrane reviews versus reviews in peer reviewed journals

Of the 10 reviews with scores of 4 or more, seven were published in the Cochrane Library. Cochrane reviews had higher overall quality scores than those published in peer reviewed journals (median 6 v 2, P<0.005). Compared with reviews published in journals, more Cochrane reviews had adequate reporting of the search strategy, had comprehensive search strategies, described the inclusion criteria, made efforts to avoid bias during the selection of the studies, reported the criteria for assessing the quality of the studies, reported the methods used to combine study findings, and combined the findings of the relevant studies appropriately (table (table2).2). Fewer Cochrane reviews had language restrictions (0/12 v 21/38, P<0.005) and more included heterogeneity testing (9 v 11, P=0.007).

There were no significant differences between the groups in the number of authors, the number of studies reviewed, assessment of publication bias, and identification of primary outcomes.

Findings of reviews

Twelve reviews identified an intervention as better than the control treatment. None of the reviews reported patient preferences. The box gives the conclusions of the reviews with minimal flaws.

Findings of studies with minimal flaws

Peer reviewed journals

  • Use of oral and intravenous steroids early in the treatment of asthma exacerbations reduces hospital admissions in adults and children; steroids are effective in preventing relapse in outpatient management of exacerbations; and oral and intravenous steroids seem to have equivalent effects on pulmonary function during acute exacerbations18
  • Addition of ipratropium bromide to β agonist significantly improves the predicted forced expiratory volume in one second but gives no clinical improvement19
  • Bronchodilator delivery through a meter dosed inhaler or wet nebuliser is equivalent in acute treatment of asthma20

Cochrane Library

  • Limited asthma education is unlikely to improve health outcomes in adults with asthma21
  • Addition of a single inhalation of anticholinergics to a β agonist regimen may improve lung function but has no notable effect on hospital admissions. Multiple doses may avoid hospital admission in 1 of 11 treated patients22
  • A short course of corticosteroids (oral or intramuscular) after an acute exacerbation of asthma (of variable and poorly reported severity at baseline) reduces the number of relapses and decreases use of β agonist without apparent increase in adverse effects23
  • There is no indication for routine use of methotrexate in patients receiving long term treatment with oral steroids24
  • Prophylaxis with single doses of nedocromil sodium1-8 mg inhibits the severity and duration of exercise induced bronchoconstriction. Benefits seem greatest in patients with more severe symptoms25
  • Allergen specific immunotherapy reduces the symptoms and drug requirements but has no consistent effect on lung function. The effects are greater in patients with allergen specific bronchial hyperreactivity than in those with non-specific hyperreactivity26
  • Chemical and physical measures to reduce exposure to mites or their products at home seem to be ineffective27

Discussion

The number of systematic reviews and meta-analyses evaluating the treatment of asthma is increasing. Most of the reviews have clinical relevance but have methodological limitations that could have been avoided easily. Future reviews should include comprehensive and well reported search strategies, attempts to reduce bias during the selection of the studies, assessment of the validity of the studies, and clear description of the methods used to synthesise the data available. Few reviews address safety issues or the effect of treatment on quality of life, costs, or patient preferences.

This study also confirms that Cochrane reviews are more rigorous than reviews published in peer reviewed journals.9,14 Several factors may explain the differences. Cochrane reviews follow standardised instructions, are peer reviewed at several levels, and receive input from different groups at different stages.13 One limitation of our analyses is that the tool used for quality assessments was developed by Andrew Oxman, who led the development of the methodology of Cochrane reviews.

Perhaps peer reviewed journals could improve the quality of the systematic reviews they publish by providing authors and peer reviewers with clear reporting criteria, including those followed by Cochrane reviewers,28,29 by influencing the review process at a much earlier stage, and by promoting more frequent updates or correction of published material on the internet. Authors should be particularly careful when doing reviews in association with industry, as such reviews have been shown to have an increased risk of producing results in favour of the interventions promoted by the sponsors.12 This was the case for all but one of the reviews associated with industry in our study. The only review that did not favour an intervention was designed to evaluate the effects of vitamin C on asthma, not a new proprietary compound.17

Bridging the gap between methodological research and methodological practice will require unprecedented efforts by researchers, consumers, funders, journal editors, and peer reviewers. Efforts made by members of the Cochrane Collaboration are already contributing to filling the gap and should be emulated by peer reviewed journals.

What is already known on this topic

Systematic reviews and meta-analyses could help decision makers cope with information overload

The number of systematic reviews and meta-analyses evaluating asthma treatments is increasing

What this study adds

Most reviews of asthma treatment published in peer reviewed journals or funded by industry have serious methodological flaws that limit their usefulness

Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals

The clinically relevant messages from the 10 most rigorous reviews are summarised

Supplementary Material

[extra: Details of search strategy]

Acknowledgments

We thank Ann Murray for editorial assistance. The review was coordinated by Foresight Consultants, a division of Foresight Links Corporation.

Footnotes

Funding: Foresight Consultants was commissioned by AstraZeneca Research and Development Lund, Sweden, and AstraZeneca US to perform an independent review of this topic through an unrestricted contract. Representatives from AstraZeneca did not participate in the study selection or appraisal or in the analysis of the data. ARJ was supported in part by a National Health Research Scholar Award from Health Canada, Ottawa, Ontario.

Competing interests: ARJ is codirector of the Canadian Cochrane Network and Centre.

website extra: Details of the search strategy and all included studies are available on the BMJ's website www.bmj.com

References

1. O'Byrne PM. Clinical comparisons of inhaler systems—what are the important aspects. J Aer Med. 1995;8:S39–S47. [PubMed]
2. Mulrow CD. Rationale for systematic reviews. In: Chalmers I, Altman DG, editors. Systematic reviews. London: BMJ Publishing; 1995. pp. 1–8.
3. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Ass J. 1988;138:687–703.
4. Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997;127:820–826. [PubMed]
5. Sacks HS, Berrier J, Reitman D, Pagano D, Chalmers T. Meta-analyses of randomized control trials: an update of the quality and methodology. In: Bailar JC III, Mosteller F, editors. Medical uses of statistics. Boston: New England Medical Journal Publications; 1992. pp. 427–442.
6. Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med. 1996;3-4:216–224. [PubMed]
7. Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM. The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA. 1995;274:1942–1948. [PubMed]
8. Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol. 1996;49:235–243. [PubMed]
9. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280:278–280. [PubMed]
10. Moher D, Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, et al. Does the poor quality of reports of randomized trials exaggerate estimates of intervention effectiveness reported in meta-analyses? Lancet. 1998;352:609–613. [PubMed]
11. Bero LA, Rennie D. Influences on the quality of published drug trials. Int J Technol Assess Health Care. 1996;12:209–237. [PubMed]
12. Barnes EC, Bero L. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279:1566–1570. [PubMed]
13. Jadad AR, Haynes RB. The Cochrane Collaboration—advances and challenges in improving evidence-based decision making. Med Decis Making. 1998;18:2–9. [PubMed]
14. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629–634. [PMC free article] [PubMed]
15. Berlin JA.for the University of Pennsylvania Meta-analysis Blinding Study Group. Does blinding of readers affect the results of meta-analyses? Lancet 1997. 350185–186.186 [PubMed]
16. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–1278. [PubMed]
17. Bielory L, Gandhi R. Asthma and vitamin C. Ann Allergy. 1994;73:89–96. [PubMed]
18. Rowe BH, Keller JL, Oxman AD. Effectiveness of steroid therapy in acute exacerbations of asthma: a meta-analysis. Am J Emerg Med. 1992;10:301–310. [PubMed]
19. Osmond MH, Klassen TP. Efficacy of ipratropium bromide in acute childhood asthma: a meta-analysis. Acad Emerg Med. 1995;2:651–656. [PubMed]
20. Turner MO, Patel A, Ginsburg S, FitzGerald JM. Bronchodilator delivery in acute airflow obstruction. A meta-analysis. Arch Intern Med. 1997;157:1736–1744. [PubMed]
21. Gibson PG, Coughlan J, Wilson AJ, Hensley MJ, Abramson M, Bauman A, et al. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. The effects of limited (information only) patient education programs on the health outcomes of adults with asthma.
22. Plotnick LH, Ducharme FM. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Efficacy and safety of combined inhaled anticholinergics and beta-s-agonists in the initial management of acute pediatric asthma.
23. Rowe BH, Spooner CH, Ducharme FM, Bretzlaff JA, Bota GW. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. The effectiveness of corticosteroids in the treatment of acute exacerbations of asthma: a meta-analysis of their effect on relapse following acute assessment.
24. Davies H, Olson L, Gibson P. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Methotrexate as a steroid sparing agent in adult asthma.
25. Spooner CH, Saunders LD, Rowe BH. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Nedocromil sodium as single dose prophylactic treatment of exercise-induced bronchoconstriction.
26. Abramson MJ, Puy RM, Weiner JM. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Allergen specific immunotherapy for asthma.
27. Hammerquist C, Burr ML, Gotzsche PC. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. House dust mite control measures in the management of asthma.
28. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF, for the QUOROM Group. Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement [Eprint, unreviewed]. Lancet Interactivewww.thelancet.com/newlancet/reg/issues/vol354no9172/commentary2.html (accessed 18 October, 1999). [PubMed]
29. Shea B, Dube C, Moher D. Assessing the quality of reports of systematic reviews and meta-analyses: a systematic review of checklists and scales. Abstract presented at the VII Cochrane Colloquium. 1999.

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Group
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...