NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Shekelle PG, Newberry SJ, Wu H, et al. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun.

Cover of Identifying Signals for Updating Systematic Reviews

Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods [Internet].

Show details

Results

This chapter presents the results of our comparisons of the RAND and Ottawa update methods for each of the three reviews in turn.

Effects of Omega-3 Fatty Acids on Cognitive Function

The literature search identified 1,009 articles from an unrestricted set of medical journals, and experts suggested an additional 5 articles, for a total of 1,014 articles whose abstracts were reviewed for relevance. Of those, 89 articles were selected for full-text review, and 26 were abstracted. Selection criteria were consistent with those used in the original review. Specific exclusion criteria were: no outcomes of interest, weak study design (i.e., cross-sectional, descriptive, or narrative review), study population <18 years old, cognitive/neurological condition was mental health disease or trauma related, and no specific measure of fat intake attributable to fatty fish/omega-3 fatty acids. Systematic reviews were reference mined but not abstracted. Since nearly all of the original studies listed in systematic reviews were already identified and abstracted from the Ottawa reference search, also abstracting the systematic reviews would have resulted in double counting of many study populations. The article flow is in Appendix A, and detailed article abstractions are in Appendix C.

Informational letters were sent to 25 experts, of whom 5 returned responses. The remaining 20 individuals cited time constraints and lack of knowledge about recent evidence in the field. The poor response rate is attributable to the small number of experts who actively follow this area of research; several of those who responded also indicated they had little knowledge about the current state of research related to omega-3 fatty acids and cognitive or neurological disorders.

Overall Results for Cognitive Function

Both the Ottawa and RAND updating signals had an absolute agreement of 100 percent and kappa of 1.0 for the 5 Key Questions related to the role of omega-3 fatty acids in cognitive and neurological disorders (see Table 2). Key Questions 1–3 were determined to be out of date, and Key Questions 4–5 were considered up to date by both methods. Appendix B includes a detailed summary of the RAND and Ottawa-based conclusions for each Key Question. Our conclusions also did not change when articles recommended by experts were excluded, although experts did recommended two new articles that did not appear in the Ottawa literature search due to their very recent publication.

Table 2. Comparison of signals for updating the review on cognitive function.

Table 2

Comparison of signals for updating the review on cognitive function.

We note that since Ottawa's updating signals were designed around the characteristics of clinical trials, and since much of the evidence for cognitive/neurological disease was from population-based studies, we used modified interpretations to determine whether evidence met an Ottawa qualitative signal for updating. Namely, none of the articles reviewed was from a major general medical journal—all were from specialty journals, so it was not possible to meet Ottawa's original definition of a pivotal trial in that respect. Also, for Key Question 2, the largest study group included in the new evidence (n=8,085) was less than three times as large as the largest study group in the original review (n=5,386). Unlike clinical trials, where the original study population might be modest (one example in this report is a sample size of 20), in large cohort studies, a threefold change in sample size may be more difficult to achieve and may not mean the same thing as a threefold change in sample size for a clinical trial. Without these modified definitions, Key Questions 1–3 would not have met strict Ottawa criteria for updating. As a group, and with input from the original developers of the Ottawa method, we discussed these modifications and believe that with them the new evidence is consistent with Ottawa's original intent and approach to signaling for updates.

In addition to the five Key Questions evaluated in the original review, two articles describing a new but related Key Question were identified—the effect of omega-3 fatty acids on treatment of Huntington's disease. While this review focused on evaluating whether existing Key Questions were still up to date, the identification of a new study question might be interpreted as a broader signal about whether the entire review is out of date.

Effects of Omega-3 Fatty Acids on Cancer

The literature search identified 125 articles from a combination of the 5 pivotal medical journals and 6 specialty journals suggested based on their rate of citation in the original review: American Journal of Clinical Nutrition, The European Journal of Clinical Nutrition, JPEN Journal of Parenteral & Enteral Nutrition, The American Journal of Public Health, The American Journal of Epidemiology, and BMC (Biomed Central) Cancer Journal. Experts suggested an additional 15 articles. In addition, a small independent search was conducted for articles that cited an article recommended by one of the experts on the topic of omega-3 fatty acids and cancer treatment, because the search design did not capture articles on this topic; this search identified 16 potentially relevant articles. Thus, abstracts for a total of 162 articles were reviewed for relevance. Of those, 67 articles were selected for full-text review, and 47 were abstracted, of which 20 were subsequently rejected. Selection criteria were consistent with those used in the original review. Specific exclusion criteria were: no outcomes of interest, weak study design (i.e., cross-sectional, descriptive, or narrative review), and no specific measure of fat intake attributable to fatty fish, fish oil, or total or specific omega-3 fatty acids. Systematic reviews were reference mined but not abstracted. Nearly all of the original studies listed in systematic reviews were already reviewed in the original review, or were identified and abstracted from the University of Ottawa Evidence-based Practice Center (UO EPC) search. The article flow is in Appendix A, and detailed article abstractions are in Appendix C.

The RAND Method

Informational letters were sent to 13 experts, of whom 7 returned responses. The remaining six individuals did not respond. Of those who responded, at least one cited a lack of knowledge about the current research in the field, and at least two provided responses for only one or two of the Key Questions. However, several recommended articles, including one that responded to Key Question 2 (for which no titles had appeared in the UO EPC literature search, apparently due to search design). A number of the other articles cited were already included, either in the original review or in the results of the search for this report.

The Ottawa Method - Quantitative Signals for Cancer

We followed methods for detecting a quantitative signal as discussed in the original Ottawa method report.5, 6 For each of the three previously pooled outcomes, the original results from the meta-analysis were pooled using a fixed effects model with data from each of the new studies.

Since there were no pivotal trials, we started with the largest trial. After each study was pooled, the results were reviewed to see if one or more of the quantitative signals were met. The quantitative signals included: (1) a change in statistical significance (must have a p-value <0.04 or >0.06), (2) a relative change in effect size of at least 50 percent, (3) a significant difference between the old and new point estimates. If one of the quantitative signals appeared as true, then no new studies were added for a given outcome. As long as a quantitative signal did appear, then studies were added until a quantitative signal was detected or there were no studies left to add.

The original review provided meta-analytic poolings for three outcomes (complications, mortality, and length of hospital stay). Of the 10 new articles identified for this report 14-23, 7 reported at least one of the previously used outcomes.15-21 Three studies reported on mortality,15, 16, 19 four studies reported on length of stay,15, 18, 20, 21 and all seven reported on complications. For the mortality outcome, the original review showed a nonsignificant odds ratio (95% confidence interval [CI]) of 1.25 (0.64, 2.48). None of the three signals was detected after adding any of the three studies that reported mortality. The original review estimated a relative risk of 0.57 (0.46, 0.72) for the complications outcome. Again, no quantitative signal was detected when adding any of the seven new trials that reported complications. The pooled mean difference (95% CI) in the original review for length of hospital stay was −3.17 (−4.12, −2.23). Adding the four new trials did not allow us to detect a quantitative signal.

Overall Results for Cancer

The Ottawa and RAND updating signals agreed for 19 of 22 Key Questions/subquestions (see Table 3), resulting in a kappa of 0.36 for all of the Key Questions related to the role of omega-3 fatty acids in preventing and treating cancer. In the sensitivity analysis, the kappa statistic was 0.62. For reference, Appendix C includes the evidence tables constructed from the literature identified in the update searches.

Table 3. Comparisons of signals for updating the review on cancer.

Table 3

Comparisons of signals for updating the review on cancer.

Appendix B includes a detailed summary of the RAND and Ottawa-based conclusions for each Key Question.

In applying the Ottawa signaling criteria to the studies in this report, a number of modifications were needed. First, the original review was able to pool results for only a small proportion of the outcomes, all of which were outcomes of trials assessing the effect of perioperative omega-3 fatty acid-supplementation of patients who underwent surgery to treat upper-gastrointestinal tract cancers (postoperative complications, length of hospital stay, and mortality). Therefore, we were able to seek a quantitative signal only for these outcomes (and found none). Second, the entire body of evidence for cancer prevention was from population-based studies, both prospective cohort and nested case control studies; therefore, we used modified interpretations of the Ottawa qualitative signals to determine whether the evidence met an Ottawa qualitative signal for updating. As a group, and with input from the original developers of the Ottawa method, we discussed these modifications and believe that with them the new evidence is consistent with Ottawa's original intent and approach to signaling for updates. Only one of the articles reviewed was from a pivotal (i.e., major general medical) journal—the remainder were from specialty journals, and that one article was one of a number of articles that presented conflicting findings, so it was not possible to meet Ottawa's original definition of a pivotal trial for any of the Key Questions.

There were three instances where the RAND method produced a signal suggesting the probable or definite need for updating for a particular question, whereas the Ottawa method did not produce a quantitative or qualitative signal for the need to update. All three were similar: In two cases there was a modest amount of evidence in the original review and in the third case there were no relevant studies in the original review; the update search identified a small number of new studies (1, 1, and 2 new studies) that did not meet Ottawa criteria for a qualitative signal but were judged by the RAND method as a signal that the original conclusion was probably or definitely out of date.

Effects of Omega-3 Fatty Acids on Cardiovascular Risk Factors

The literature search identified 613 articles. We screened the abstracts using the same eligibility criteria as for the original review. We included the cardiovascular risk factors and the outcome-specific study design features listed in Table 3.1 (page 35) of the original review.11 We included only studies with omega-3 interventions (diet or otherwise) less than or equal to 6 g/day. We also included narrative and systematic reviews if they were relevant to this topic.

Based on the sources of studies in the original review, in addition to the five “pivotal” journals, we searched the American Journal of Cardiology; the American Journal of Clinical Nutrition; the American Journal of Medicine; Arteriosclerosis, Thrombosis, and Vascular Biology; Atherosclerosis; the British Journal of Nutrition; Cardiovascular Research; Diabetes Care; Diabetologia; the European Heart Journal; the European Journal of Clinical Nutrition; the Journal of Nutrition; Lipids; Thrombosis and Haemostasis; Thrombosis Research; and Vascular Medicine.

After full-text screening, data from 60 articles (6 of which were systematic reviews) were then extracted. In addition, 31 articles were later identified by expert opinion, 5 of which had already been accounted for. After screening the remaining suggested articles, 12 of these were ultimately included (for evaluation of the RAND method only).

In total, 15 experts were contacted to provide their input for this topic. These included all of the peer reviewers and Technical Expert Panel members of the 2004 review, as well as other topic experts. Of these, six replied with the completed form as well as a completed disclosure of interest form.

Ottawa Method

Quantitative Signals

Based on the original review and accompanying journal articles,24, 25 there were nine outcomes for which meta-analyses were reported. The meta-analyses evaluated the effect of fish oils on total cholesterol, low-density lipoprotein cholesterol (LDL), high-density lipoprotein cholesterol (HDL), triglycerides, systolic and diastolic blood pressure (in people without diabetes mellitus), hemoglobin A1c, fasting glucose, and coronary artery restenosis by arteriography. New trial evidence was found for all outcomes except coronary artery restenosis. A very large trial from Japan compared eicosapentaenoic acid supplementation with placebo in 18,645 participants (4,565 of whom had impaired glucose metabolism [Japan Eicosapentaenoic Acid Lipid Intervention Study (JELIS)]).26 This trial found discordant results with almost all other trials (both old and new trials) for several outcomes. Thus, a quantitative signal was found after adding this single trial to the meta-analyses for total cholesterol (from a null effect to a statistically significant net reduction with fish oil), LDL (from a statistically significant net increase with fish oil to a null effect), HDL (from a statistically significant net increase with fish oil to a null effect), systolic and diastolic blood pressures (from statistically significant net decreases with fish oil to a null effect on systolic blood pressure and a statistically significant increase in diastolic blood pressure), and fasting glucose (from a nonsignificant effect to a smaller, but statistically significant, net increase with fish oil). No quantitative signal was found for triglycerides after the addition of 10 new trials. No quantitative signal was found for hemoglobin A1c after the addition of four new trials.

Of note, however, due to incomplete reporting, we had to estimate the standard errors of the net differences from JELIS. Given the very large size of the trial, and thus the very small standard errors, we ended up estimating P values of the net differences that were discordant with the study conclusions. This impacted the meta-analyses. The study reported no significant effect on diastolic blood pressure, however our estimate yielded a very small but statistically significant net increase (1 mm Hg; 95% CI 0.7, 1.3), which resulted in a statistically significant summary estimate. A similar set of discordant results occurred with fasting glucose.

Qualitative Signals

Fish Oil

The numbers of pivotal trials identified for total, LDL, and HDL cholesterols, and blood pressure were 5, 4, 6, and 7, respectively. Among these pivotal trials, one very large trial (JELIS, N = 18,645) reported findings that disagreed with the results from the original review for total, LDL, and HDL cholesterols, and systolic and diastolic blood pressure in people with diabetes, and therefore triggered qualitative signals for these outcomes.

Pivotal trials were identified for triglycerides (three studies), lipoprotein (a) (five studies), apolipoprotein A-1 (one study), apolipoprotein B (two studies), blood pressure in people without diabetes (six studies), hemoglobin A1c (four studies), fasting glucose (five studies), fasting insulin (nine studies), C-reactive protein (CRP) (five studies), fibrinogen (three studies), factor VII (three studies), non-Willebrand factor (three studies), platelet aggregation (one study), carotid intima media thickness (one study), but the findings of these trials did not oppose the results of the original review. No pivotal trials were identified for apolipoprotein B-100, LDL apolipoprotein B, factor VIII, coronary arteriography, exercise tolerance testing, and heart rate variability. Therefore, there were no qualitative signals for these outcomes.

Alpha-Linolenic Acid (ALA)

Qualitative signals were found for total cholesterol and LDL cholesterol, because pivotal new trials reported opposing findings. Two new trials of ALA found no effect on total cholesterol and LDL cholesterol, in contrast to the four trials in the original review which generally found small net increases in the lipids.

One pivotal trial was identified for fibrinogen, but the findings of this trial did not oppose the results of the original review. No pivotal trials were identified for the remaining cardiovascular risk factors. Therefore, there were no qualitative signals for these outcomes. However, among these outcomes, the original review included no evidence on the effect of ALA on seven outcomes (systolic and diastolic blood pressure, hemoglobin A1c, fasting glucose, fasting insulin, CRP, and carotid intima media thickness), and one or two new but small trials were found in the updated literature search. As a sensitivity analysis, these were categorized as having an A4 signal, that there were important changes in the effectiveness short of “opposing findings.”

RAND Method

Fish Oil

The RAND method produced a signal indicating the probable need to update the original review for heart rate variability based on new trials and a minority of experts' opinion. The original review suggested no significant effect of fish oil on heart rate variability among healthy volunteers, but that heart rate variability may increase in patients with myocardial infarction. However, three of six new trials found that the fish oil group significantly improved heart rate variability. (Three of these studies were suggested by experts and were therefore not included in the Ottawa method.)

The RAND method showed a signal for the possible need to update total cholesterol and CRP based on new trials. The original review found that fish oil resulted in small nonsignificant net increases in total cholesterol and CRP. New studies, however, showed the fish oil groups had a significant decrease in total cholesterol and CRP. Based on a minority of experts' opinions, the RAND method found the possible need to update fasting glucose, systolic and diastolic blood pressure (in people with or without diabetes), fasting insulin, and coronary arteriography. One out of six experts indicated that the conclusions from the original review are out of date for systolic and diastolic blood pressure. However, one new systematic review27 provided by the expert found very similar net changes as determined in the original review. Similarly, the findings on fasting glucose and fasting insulin were determined to be consistent overall between the original review and the new evidence. Lastly, one expert stated that the findings on coronary arteriography from the previous systematic review are out of date without providing any new evidence.

The RAND method did not produce any signal indicating the need for updating LDL or HDL cholesterol, triglycerides, lipoprotein (a), apolipoprotein A-1, apolipoprotein B, apolipoprotein B-100, LDL apolipoprotein B, hemoglobin A1c, fibrinogen, factor VII, factor VIII, von Willebrand factor, platelet aggregation, carotid intima media thickness, and exercise tolerance testing.

Alpha-Linolenic Acid (ALA)

The RAND method indicated a signal to possibly update total cholesterol, apolipoprotein A-1, and apolipoprotein B based on new studies, which had some evidence of significant decreases in these outcomes with ALA. In contrast, the original review found that ALA intake resulted in small or nonsignificant effects on these outcomes. Additionally, the RAND method demonstrated a signal for the possible update for LDL because the ALA groups showed net increases in LDL in the original review, but no effects on LDL from the updated search. We also concluded that there is a signal to probably update systolic and diastolic blood pressure, hemoglobin A1c, fasting glucose, fasting insulin, C-reactive protein, and carotid intima-media thickness because there were no ALA studies in the original review, but we found one or two new trials for these outcomes (three for C-reactive protein) from the updated search and the articles provided by the experts.

The RAND method demonstrated that the conclusions of the previous SR are still valid (based on new studies and experts' opinions) for HDL cholesterol, triglycerides, lipoprotein (a), apolipoprotein B-100, LDL apolipoprotein B, fibrinogen, factor VII, factor VIII, von Willebrand factor, platelet aggregation, coronary arteriography, exercise tolerance testing, and heart rate variability.

Summary of Discordant Results

Fish Oil

We obtained discordant results for 8 of 26 intermediate outcomes in studies of fish oil and cardiovascular risk factors when comparing between the RAND and Ottawa methods (Table 4; Appendix Table B4). The Ottawa method determined that there was no signal for systolic and diastolic blood pressure among diabetic patients, fasting insulin, CRP, coronary arteriography, and heart rate variability due to the absence of a pivotal trial. However, the RAND method found that results from the previous systematic review was possibly out of date for CRP based on new trials; probably out of date for heart rate variability based on expert opinion, one new trial found by the literature search, and three new trials from the experts; and possibly out of date for blood pressure in people with diabetes, fasting insulin, and coronary arteriography based on experts' opinion. However, the experts' opinions were not substantiated by new trials suggested by them for these latter outcomes. The literature search found no new studies on heart rate variability that provided a signal; however, one expert said the original report was out of date and provided three new trials to support that conclusion; these three trials were not considered for the Ottawa method.

Table 4. Comparison of signals for updating the report on fish oil and cardiovascular risk factors.

Table 4

Comparison of signals for updating the report on fish oil and cardiovascular risk factors.

For LDL and HDL, we found quantitative update signals using the Ottawa method due to one large Japanese study (n=18,645). This one outlier study changed the meta-analysis results for these intermediate outcomes. However, using the RAND method, we determined that results from the previous systematic review were still valid, though with a caveat about this one outlier study. We determined that this one outlier study, though extremely large, was not sufficient to contradict the 20 to 30 other (old and new) trials.

Alpha Linolenic Acid (ALA)

We obtained discordant results for 9 of 24 intermediate outcomes in studies of ALA and cardiovascular risk factors when comparing between the RAND and Ottawa methods (Table 5; Appendix Table B5). The original review included two ALA trials with data on apolipoproteins A-1 and B. The updated literature search found one new trial that may contradict the original review's trials; however, this trial was not large enough to be pivotal. Thus, by the RAND method, the original review is possibly out of date, but there was no signal using the Ottawa method.

Table 5. Comparison of signals for updating the report on ALA and cardiovascular risk factors.

Table 5

Comparison of signals for updating the report on ALA and cardiovascular risk factors.

There were no studies in the original review that assessed the effects of ALA on blood pressure, hemoglobin A1c, fasting glucose, fasting insulin, CRP, and carotid intima media thickness. Because the literature searches found one or two new trials for each of these outcomes (and an expert gave us a third CRP trial), the RAND method finds the previous conclusions “probably out of date.” Since none of the new findings came from pivotal studies and no meta-analyses were performed previously for ALA on these outcomes, there is no signal using the Ottawa method.

However, if we apply the approach of assigning these as A4 signals with the Ottawa method, then these 7 outcomes are concordant and there are discordant results for only 2 of 24 intermediate outcomes in studies of ALA and cardiovascular risk factors.

Comparison of Results Using the RAND and Ottawa Methods

We combined all the RAND signals, “definitely,” “probably,” and “possibly,” out of date into a single category, “out of date” and compared the conclusions between methods within clinical topic area and across topics (Tables 610). The range of agreement varied from a kappa of 0.19 to “almost perfect” agreement with a kappa of 1.0. Overall across all 77 conclusions, agreement was classified as “fair.”13

Table 6. Overall comparison and kappa statistic for cognitive function.

Table 6

Overall comparison and kappa statistic for cognitive function.

Table 7. Overall comparison and kappa statistic for cancer.

Table 7

Overall comparison and kappa statistic for cancer.

Table 8. Overall comparison and kappa statistic for cardiovascular disease markers: fish oil.

Table 8

Overall comparison and kappa statistic for cardiovascular disease markers: fish oil.

Table 9. Overall comparison and kappa statistic for cardiovascular disease markers: ALA.

Table 9

Overall comparison and kappa statistic for cardiovascular disease markers: ALA.

Table 10. Overall comparison and kappa statistic for all reviews.

Table 10

Overall comparison and kappa statistic for all reviews.

As discussed previously, many of the disagreements between methods were due to a situation where the original review had a Key Question with no evidence, and some evidence was identified in the update. In these situations, the RAND method produced a positive signal for updating and Ottawa's method produced a negative signal. Reclassifying these situations as agreement between the two methods yields much better estimates of agreement; for three of the four conditions, agreement was “substantial” to “almost perfect,” and overall agreement was “substantial.”

PubReader format: click here to try

Views

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...