NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Raman G, Gaylor JM, Rao M, et al. Quality of Reporting in Systematic Reviews of Implantable Medical Devices [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Nov.

Cover of Quality of Reporting in Systematic Reviews of Implantable Medical Devices

Quality of Reporting in Systematic Reviews of Implantable Medical Devices [Internet].

Show details


We identified 30 items from the PRISMA and MOOSE checklists that were relevant to our Key Questions,5,6 along with 8 new device-specific and operator-specific items (see final list in Table 1). To our knowledge, there has been no prior empirical evaluation of systematic reviews of implantable medical devices. Although the deficiencies in reporting regarding some of items in systematic reviews of implantable medical devices are similar to those seen in reviews of drug-therapy studies,17,18 our findings highlight types of deficiencies that should be remedied. In particular, reviewers should a priori adhere to a specific guideline (e.g., those described in this report) when conducting a systematic review in order to avoid neglecting to report relevant characteristics within primary studies. Secondly, when conducting a review (and transitively, a primary study), it is essential that variation within the intervention with potential to influence or confound outcomes is reported or at least identified and acknowledged as a possible limitation. Consequently, heterogeneity should be adequately evaluated through subgroup or sensitivity analyses.

Our analyses of a large sample of systematic reviews of implantable medical devices found that about 20 of 30 recommended items were commonly reported (as defined by reporting in at least 50 percent of the reviews). However, eight items identified as specific to the field of medical devices were all infrequently reported across device categories. We also identified inadequate reporting of 9 of 17 items that represented the clarity or transparency of methods and results. There were no significant differences in quality of reporting when reviews were stratified by their reporting of authors' affiliations (vs. not reporting affiliations) to industry, except for one item of conducting a meta-analysis. It is possible that journals mandating disclosure of authors' affiliation were more likely to accept systematic reviews with a meta-analysis than those without a meta-analysis.

Our review also shows that the majority of the meta-analyses were conducted by applying accepted methodologies (92 percent). The remaining eight percent of the meta-analyses were performed in the presence of heterogeneity among device groups (e.g., by combining data across drug-eluting stents and bare-metal stents) or among studies combined across designs (e.g., by combining data across study designs of randomized trials and observational studies). In the presence of such heterogeneity, and by combining such studies into meta-analyses, the meaning of the result is unclear. For example, the utility of assessing outcomes of studies confounded by type of drug-eluting stent or performance of different operators at different sites is unclear.

The number of systematic reviews of implantable medical devices has grown rapidly in recent years, with reviews being published in a broad range of journals. Our results indicate that current systematic reviews of implantable medical devices generally lack data on the reporting of some important items applicable to any systematic review, as well as data relevant to device- and operator-specific items that are specific to review of implantable medical devices. The device-specific factors—including evolution of technology, generalization of results from one device to a similar device, evaluation of device–operator interactions, and evaluation of team expertise—are important characteristics that should be examined in systematic reviews of implantable medical devices. Since there is no widely accepted guidance for reporting of information unique to implantable medical device studies, failure to report data on procedures and devices could potentially lead to invalid synthesis or interpretation of results.

Failure to report the variation in device specifics and operator techniques can potentially confound results. For example, several reviews of stent studies combined sirolimus- and paclitaxel-eluting stents together into a generic category of “drug-eluting stents” when compared to bare-metal stents, without additional subgroup analyses. This highlights a need for the identification and inclusion of items to address device-specific information in a systematic review. The lack of reporting of these potentially important variables may stem from the fact that most reviews focused on evaluating clinical outcomes rather than whether device- or operator-specific variables influenced the clinical outcomes. While this critical appraisal was under peer review, we evaluated 100 primary studies for quality of reporting of device- or operator-specific details (detailed methodology in Appendix F). Our review identified that primary studies frequently report device-specific data but infrequently report operator-specific data (Appendix F). Our findings do emphasize the need for improving quality of reporting in both primary studies and systematic reviews.

Systematic reviews have gained acceptance as a useful way to summarize data and are also helpful in identifying knowledge gaps within primary studies as well as reviews of those studies. Findings from systematic reviews can help target current and identify future specific research needs. Therefore, good-quality reporting and well-conducted systematic reviews can minimize the likelihood of bias or misinterpretation of results. Systematic reviews and meta-analyses represent a very high level within a hierarchy of evidence, making it all the more important that they are conducted as methodologically rigorous as possible.


The quality of reporting within the available primary literature is always a limitation, but these limitations ought to be systematically acknowledged and managed. Some of the generic items were observed to be reported in 100 percent of the reviews, while other items were rarely reported. Our examination relied on reporting by the authors of these reviews. It is possible that the authors of these reviews conducted comprehensive evaluations but we cannot comment on this unless authors clearly report their methods in journal publications. We did not check for data-extraction errors within the systematic reviews, contact review authors, or conduct any reanalysis of primary data from those reviews, as none of these was the primary purpose of our review. We did not conduct searches of grey literature to identify unpublished data. Thus, we cannot comment on the quality of reporting in grey literature. Our TEP panel did not include clinical experts. While inclusion of clinical experts as TEP members was desirable, it would have required a large panel of members (one for each device category) and was therefore not pursued. Finally, we used liberal criteria for reporting an item (either in the analyses or descriptively in the discussion). By using a very low threshold for reporting of device- and operator-specific information, our results may have inflated the numbers with regard to reporting of these important variables.

Recommendations of Reporting Items for Systematic Reviews of Implantable Medical Devices

Reporting of Device- or Procedure-Specific Data

Our report indicates that the reporting of study characteristics with regard to device- or procedure-specific data needs to be improved. Differences within devices or across device groups were reported in less than half of the reviews and less than one-tenth of the systematic reviews providing information on operator- or procedure-specific data. It is unclear whether the space allotted or word count of the journal is the reason for this poor reporting; if so, journal editors and reviewers should encourage authors to provide supplementary material for posting on a Web page. Information that should be reported includes the device characteristics, evolution of the device during the study period, details of training or certification of the operator, the operator learning curve, the level of expertise of the operating team or site, variations among practitioners, and volume at the sites that conducted the study.

In addition, PROSPERO—the International Prospective Register of Systematic Reviews—provides guidance on reporting, conduct, scientific writing, and publication of systematic reviews through a formal protocol registration process. We suggest incorporation of eight new device- and operator-specific items unique to implantable medical device studies at the time of registration of systematic reviews' protocols. Such an effort would encourage researchers to practice accurate and transparent reporting of systematic reviews of implantable medical devices. We also suggest incorporation of eight new device- and operator-specific items unique to implantable medical device studies into the extension guidelines of PRISMA.

Herein, we describe a list of device-specific characteristics within each of the five device categories that could be considered in future systematic reviews of implantable medical devices. The characteristics in this list are in no particular order but are limited to those that were described in systematic reviews of implantable devices evaluated in this report. Our TEP panel did not include clinical experts, and therefore, to identify additional device-specific characteristics, we encourage review authors to consult domain experts before embarking on future systematic reviews.

Cardiac defibrillators with or without pacemakers:

  • Device type
  • Method of implantation
  • Position of the electrode
  • Description of microprocessor technology and programmable features
  • Alert features that monitor lead impedance

Vascular interventional devices (e.g., stents)

  • Type of stent and stenting technique
  • Generation of stent (e.g., first or second generation)
  • Type of antiproliferative drug used
  • Delivery system
  • Polymer layer
  • Stent frame

Orthopedic implants

  • Type of device
  • Surgical technique or approach
  • Number and location of devices
  • Fixation and supplementary materials such as plates and screws
  • Type of device coating

Skin-replacement grafts

  • Type of skin graft required
  • Composition of graft
  • Graft type: bioabsorbable or requiring removal


  • Stimulation parameters
    • frequency
    • intensity
    • pulse width
  • Electrode location

Reporting of Generic Items as Suggested in the PRISMA and the MOOSE Statements

Expert panels have identified guidelines for the conduct and reporting of systematic reviews resulting in statements such as the PRISMA and the MOOSE. These statements have been adopted by many major journals as a tool to ensure appropriate conduct and reporting standards for systematic reviews. Journal editors and reviewers should encourage publication of systematic reviews of implantable medical devices that adhere to the conduct and reporting standards as outlined in these statements.

Systematic Reviews of Implantable Devices Should Clearly State the Objective or Rationale For Conducting a Review

We found that the objective or rationale for conducting a systematic review of implantable medical devices was often not stated clearly. A systematic review is often conducted to confirm a result from a primary study, or a meta-analysis may be conducted to increase the sample size and to determine whether the result from a primary study holds in other populations when combined with evidence from other studies. Another objective may be to evaluate sources of heterogeneity. Without stating an objective it is often difficult for readers to understand the exact reason for which a systematic review was conducted or whether the new review would add any new information to existing knowledge base.

Systematic Reviews of Implantable Devices Should Explicitly Report Search and Study-Selection Criteria

The majority of reviews of implantable medical devices explicitly reported search criteria and selection of studies. However, only a few conducted searches in languages other than English or attempted to include unpublished data. Moreover, in our review, less than one-half of the reviews reported the numbers of papers identified using a flow diagram, as is suggested in the PRISMA and the MOOSE statements. As compared with published trials, unpublished trials tend to show less beneficial effect, but non–English-language trials and nonindexed trials tend to show larger treatment effects.19 Therefore, emphasis should be placed on identifying all available evidence by performing a comprehensive literature search (including unpublished studies and non–English-language studies). In addition, comparing unpublished data with published data can be useful in evaluating the potential impact of publication bias.

Systematic Reviews of Implantable Devices Should Explore Heterogeneity Through Subgroup and Sensitivity Analyses

Systematic reviews often explore the degree to which data from individual studies (e.g., from sensitivity analyses) or any variation in relation to specific clinical characteristics of the included studies (e.g., from subgroup analyses) affect the main findings. Our report shows that only half of the reviews used subgroup analyses and used sensitivity analyses to test whether the results of their review are robust. Authors usually perform a variety of analyses and they should publish all their analyses.

Systematic Reviews of Implantable Devices Should Assess the Risk of Bias of the Primary Studies Included

In our review, only 40 percent of systematic reviews of implantable medical devices assessed the risk of bias or used quality scales or checklists to assess the methodological quality of the primary studies included. Without these assessments, the internal validity of the included primary studies is unknown and therefore the impact of the potential biases in the primary studies on the conclusions of a systematic review remains unclear. Furthermore, transparent reporting of the risk of bias ensures more accurate, less biased summaries of the overall evidence that allow users of the systematic reviews to have a better understanding of the summarized evidence and what biases may exist.

Systematic Reviews of Implantable Devices Should List Funding Sources and Authors' Conflicts of Interest as Part of Their Standard Reporting

We found that only 42 percent of systematic reviews reported the funding source, and about 20 percent reported authors' financial affiliation to industry. Some empirical evidence from drug-therapy trials has shown an association between the reporting of favorable results and industry funding and financial ties between authors and industry. Systematic reviews of implantable medical devices should, as part of their standard reporting, disclose device industry funding and authors' affiliation with the device industry.

Systematic Reviews of Implantable Devices Should Formally Assess the Overall Body of Evidence

Only 18 percent of the reviews assessed the overall body of evidence. Rating or evaluating the overall body of evidence allows systematic reviews to link the quality of the overall evidence to the strength of their conclusions. A formal rating system such as the GRADE (Grades of Recommendation Assessment, Development, and Evaluation) allows systematic reviewers to carefully examine the benefits and harms and draw reasoned conclusions by considering the uncertainty of efficacy or effectiveness of an intervention of interest. Formal assessment of the overall body of evidence is a relatively new step in systematic reviews, and the proportion of studies performing such an assessment may increase as the methods around it are adopted more widely.

PubReader format: click here to try


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...