Characteristics of Studies of CEA for Symptomatic Stenosis
We included 111 studies of CEA for symptomatic stenosis. Nine were randomized trials. Of 102 observational studies, 16 were population-based (attempted to assess all patients undergoing CEA in a pre-defined population). Eight of the 111 studies were published in journals with a high (>7) Journal Impact Factor. Eighty-two studies were performed in North America and 23 in Europe. Ten had a single surgeon author, 53 had multiple surgeon authors, and 48 had at least one non-surgeon author. Many studies did not report cardiovascular risk factors. For example, the proportion of patients with diabetes, the most frequently reported co-morbidity, was reported in 57 percent of studies.
Tests of Association
Three quality criteria were very highly (p<0.001) associated with one another (Table 2): criterion 4 (adverse events pre-defined), criterion 5 (ascertainment technique adequate described), and criterion 6 (non-biased ascertainment). Overall quality rating (good, fair, or poor) was highly associated (p<0.0001) with author category (single surgeon, multiple surgeons, or at least one non-surgeon), funding source (mostly government, not reported, or other), and high (>7) Journal Impact Factor.
For five of the eight individual quality items (non-biased selection, low loss to follow-up, adverse events pre-specified and defined, ascertainment technique adequately described, and non-biased and accurate ascertainment of adverse events), meeting the criterion was significantly (p<0.05) associated with differences in rates of stroke or death (Table 3). Criteria that were not associated with significant differences in estimates of stroke or death were criterion 2 (adequate description of population), criterion 7 (statistical analysis of confounders), and criterion 8 (adequate duration of follow-up).
Other variables associated with significantly higher rates of stroke or death were author category (studies with multiple authors and at least one non-surgeon having the highest rates), prospective study, population-based study, randomized controlled trial, high Journal Impact Factor, analysis of surgeon-related variables, and the patient population variables mean age and the proportion with coronary artery disease (Tables 3 and 4).
Developing a Quality Rating Instrument
Using all eight quality criteria, reported rates of stroke or death were highest for studies rated good-quality (pooled rate 6.3 percent, 95% CI 4.6 percent to 8.0 percent), intermediate for fair-quality studies (5.3 percent, 95% CI 4.1 percent to 6.5 percent), and lowest for poor-quality studies (3.8 percent, 95% CI 3.2 percent to 4.4 percent); the differences were statistically significant for good versus poor (p=0.0076) and fair versus poor (p=0.0289), though not for good versus fair studies (p=0.3557).
To develop a more parsimonious quality rating instrument, we removed criterion 2 (adequate description of population), which was not associated with differential adverse event rates, and criteria 5 and 6, which were highly associated with (but not as predictive as) criterion 4 (adverse events pre-defined). After comparing all possible 4-item instruments using the remaining 5 criteria, we found that a quality rating instrument with criteria 1 (non-biased selection), 3 (low loss to follow-up), 4 (adverse events pre-defined), and 8 (adequate duration of follow-up) performed similarly to the 5-criteria instrument (Table 5). We also found that quality rating instruments using a single cutoff (adequate or inadequate) performed similarly to instruments using multiple cutoffs (good, fair, or poor). Using the 4-criteria instrument, studies rated “adequate” (score >=3 out of 4) had a significantly (p=0.003) higher pooled rate of stroke or death of 5.7 percent (95% CI 4.8 percent to 6.6 percent) compared to 3.7 percent (95% CI 3.1 percent to 4.3 percent) for studies rated “inadequate” (score <3). The summary quality rating also predicted rates of surgical complications better than any individual quality criterion.
For the four criteria instrument, a steady increase in the pooled complication rate was observed with increasing quality scores (scores of 1, 2, 3, and 4 associated with rates of 3.1 percent [95% CI 2.2 percent to 3.9 percent], 4.0 percent [95% CI 3.2 percent to 4.8 percent], 5.6 percent [4.4 percent to 6.7 percent], and 6.0 percent [4.5 percent to 7.4 percent], respectively). For instruments with more criteria, higher scores were associated with higher pooled complication rates until a threshold was reached at scores >=4 (Table 6).
Inter Rater Reliability
The overall quality rating (adequate or inadequate) using the four-criteria instrument was in agreement between two investigators for 19/20 of a random selection of studies (Kappa = 0.90).
Multivariate Regression Analyses
A model with four individual quality criteria (1, 3, 4, and 8) performed similarly to a model using the composite four-criteria quality rating instrument that categorized studies as “adequate” or “inadequate” (AICC goodness-of-fit test 751.4 vs. 750.7). An “adequate” rating using the four-item instrument was an independent predictor of the reported adverse event rate after adjustment for North American setting (associated with lower complication rates), randomized controlled trial or population-based study design (each associated with higher complication rates), and the proportion of total text devoted to reporting adverse event results (higher proportion associated with higher complication rates) (Table 7). The four-criteria instrument performed similarly as a binary (adequate versus inadequate) or continuous (quality score 0–4) variable (AICC goodness of fit statistic 738.1 versus 738.6).
Adding high (>7) Journal Impact Factor or the variable “single surgeon author” to the multivariate model did not improve the model. No other variables were significant in multivariate models, though incomplete reporting in the primary studies limited analysis of demographic factors and comorbidities.
An “adequate” quality rating (using the four-criteria instrument) was associated with higher reported complication rates across most clinical subgroups of patients, including patients with transient ischemic attacks (5.3 percent vs. 3.6 percent, p=0.0783), patients undergoing early (<3 to 6 weeks after symptoms) CEA (9.7 percent vs. 3.1 percent, p=0.0053), asymptomatic patients (reported separately from patients with symptoms) (3.2 percent vs. 1.4 percent, p=0.0021), and stroke patients (6.8 percent vs. 5.9 percent, p=0.3681). In multivariate analyses limited to the studies that reported mean age or proportion of patients with coronary artery disease, an “adequate” quality rating remained significant when controlling for those factors (Table 8). An “adequate” rating also predicted higher complication rates in univariate analyses of population-based observational studies (7.1 percent vs. 4.4 percent, p=0.0424) and non-population based observational studies (5.0 percent vs. 3.4 percent, p=0.0071), as well as in a multivariate model using data from all observational studies (excluding randomized trials).
Studies of CEA for Asymptomatic Stenosis
We included 18 studies of CEA for asymptomatic stenosis. Six were randomized trials, 15 were set in North American, and none were population-based studies. Rates of stroke and death were low in these studies, ranging from 0 percent to 4.6 percent.
None of the eight quality criteria (Table 9), quality rating instruments, or other clinical or study-design related variables were associated with significant differences in rates of stroke or death. Using the final four-criteria instrument developed based on the analyses of studies of CEA for symptomatic stenosis, rates of stroke or death were 2.4 percent and 2.5 percent in studies rated adequate and inadequate, respectively. In fact, little correlation was observed between quality scores and complication rates. For example, 7seven studies that only met one or two of the eight quality criteria reported complication rates ranging from 1.4 percent to 4.6 percent. Two studies that met seven or eight of the quality criteria reported complication rates of 1.3 percent and 3.0 percent.
Because no variables were significant in univariate analyses, we did not attempt multivariate regression analyses.
Studies of Rofecoxib
We included 16 randomized controlled trials of rofecoxib in patients with arthritis. Two studies published only as abstracts were excluded.36, 37 Two trials were published in journals with high (>7) Journal Impact Factor, four were published since 2001, and two were written by authors from a single department and institution. Pharmaceutical companies funded all of the trials.
In univariate analyses of the eight quality criteria, only criteria 6 (blinded external endpoint committee; pooled OR 3.69 versus 0.68, p<0.0001) and 7 (adequate statistical analysis of confounders; pooled OR 4.99 vs. 1.39, p=0.0164) were significantly associated with a higher risk for myocardial infarction (Table 10). Studies published in high Journal Impact Factor journals were the same as the studies that met criterion 7. Presence of an external endpoint committee blinded to treatment allocation was a stronger predictor of differences in risk for myocardial infarction when its presence was assessed using published or unpublished data than when relying only on published information (pooled OR 3.83 vs. 1.37, p=0.0468). Blinded assessment of complications (defined as assessor not aware of treatment allocation, but not necessarily a review committee independent from the study) was not associated with a significantly lower risk of myocardial infarction (pooled OR 1.68 vs. 3.94, p=0.2134).
Using the four criteria quality rating instrument from the analyses of studies of CEA for symptomatic stenosis, quality rating (adequate or inadequate) was not associated with significant differences in odds ratios for myocardial infarction with rofecoxib relative to control interventions (2.59 versus 1.49, p=0.28) or with absolute rates of myocardial infarction on rofecoxib (0.34 percent versus 0.31 percent).
Analyses of this data set were limited by the small number of studies, and by the fact that two studies38, 39 with high odds ratios (4.98 and 5.00) for myocardial infarction contributed over half of the patients (13633 of 23725) in the included trials. Both of these trials also met seven out of the eight quality criteria.
Agency for Healthcare Research and Quality (US), Rockville (MD)
Chou R, Fu R, Carson S, et al. Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006 Oct. (Technical Reviews, No. 13.) 3, Results.