Reporting of article retractions in bibliographic databases and online journals.

Retraction of journal papers containing falsified data plays a central role in “correcting” the scientific record. Of 395 articles retracted between 1982 and 2002, Nath et al. classified 62% as retracted as a result of mistake, 27% because of deliberate falsification or fabrication of data, and 11% without a clear reason for retraction [1]. There is evidence that retracted papers continue to be cited without reference to the retraction: Budd et al. found that 235 retracted articles were cited 2,034 times after retraction notices. Examination of 299 of these citations showed that only 19 referred to the retraction [2]. Similarly, Neale et al.'s analysis of 102 articles from PubMed showed that many of these papers were subsequently cited by other researchers who were unaware that the papers had been affected by scientific misconduct [3]. Limitations in the processes for recording retractions have been noted elsewhere, and the responsibilities of authors, journal editors, and research institutions have been highlighted [4, 5]. 
 
Following the Poehlman case in 2005, where ten articles based on fabricated data by that author were retracted, Sox and Rennie identified several opportunities for publishers to prevent citation of retracted papers [5]. These included journal editors checking the reference lists of submitted manuscripts on the National Library of Medicine website, retraction notices being linked to retracted articles on journal websites, retraction notices being prominently displayed in electronic journals, and free access to the full text of retraction notices being facilitated. 
 
While undertaking a systematic review on post-surgery analgesia [6], the authors realized that several of the papers identified for possible inclusion were by Scott S. Reuben, who was at that point being investigated for producing papers based on falsified data [7]. Twenty-one papers or abstracts authored by Reuben were under question at that time, and there was considerable publicity, while official retractions began to be made by journals. Although the presence of this publicity made it easy to weed out those articles from the systematic review, the situation offered an opportunity to conduct a case study to investigate whether databases and journals record notices of article retractions in a way that ensures they can be easily identified, even in the absence of such public discussion.


INTRODUCTION
Retraction of journal papers containing falsified data plays a central role in ''correcting'' the scientific record. Of 395 articles retracted between 1982 and 2002, Nath et al. classified 62% as retracted as a result of mistake, 27% because of deliberate falsification or fabrication of data, and 11% without a clear reason for retraction [1]. There is evidence that retracted papers continue to be cited without reference to the retraction: Budd et al. found that 235 retracted articles were cited 2,034 times after retraction notices. Examination of 299 of these citations showed that only 19 referred to the retraction [2]. Similarly, Neale et al.'s analysis of 102 articles from PubMed showed that many of these papers were subsequently cited by other researchers who were unaware that the papers had been affected by scientific misconduct [3]. Limitations in the processes for recording retractions have been noted elsewhere, and the responsibilities of authors, journal editors, and research institutions have been highlighted [4,5].
Following the Poehlman case in 2005, where ten articles based on fabricated data by that author were retracted, Sox and Rennie identified several opportunities for publishers to prevent citation of retracted papers [5]. These included journal editors checking the reference lists of submitted manuscripts on the National Library of Medicine website, retraction notices being linked to retracted articles on journal websites, retraction notices being prominently displayed in electronic journals, and free access to the full text of retraction notices being facilitated.
While undertaking a systematic review on postsurgery analgesia [6], the authors realized that several of the papers identified for possible inclusion were by Scott S. Reuben, who was at that point being investigated for producing papers based on falsified data [7]. Twenty-one papers or abstracts authored by Reuben were under question at that time, and there was considerable publicity, while official retractions began to be made by journals. Although the presence of this publicity made it easy to weed out those articles from the systematic review, the situation offered an opportunity to conduct a case study to investigate whether databases and journals record notices of article retractions in a way that ensures they can be easily identified, even in the absence of such public discussion.

METHOD
The study had two components: (1) to investigate whether articles by Reuben that had definitely been retracted at the time of this study were clearly identified as retracted in MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials (CENTRAL), the three databases considered the most important to search to identify reports of randomized controlled trials [8] and (2) to investigate whether journals that had retracted the papers displayed this information clearly and prominently.

Recording of retracted articles on bibliographic databases
The retraction notices of papers authored by Reuben first began to be published in March 2009, and the research in this study was undertaken in August 2009, by which time a substantial number of papers had been retracted. This allowed a reasonable period for the articles to have been annotated as retracted in the databases. Using a list of twenty-one articles that were under question [9] as a starting point, retraction notices were searched for to confirm that eighteen articles had definitely been retracted by that time (i.e., a notice of retraction in the journal definitely published before August 30).
MEDLINE (August Week 4 MEDLINE), EMBASE (OvidSP interface; Week 35), and CENTRAL (30 August 2009) were searched for articles by Reuben. Initial searches were for Reuben as author, as he was first author on all the papers that had been retracted. To allow for differences in how retractions are noted on EMBASE, additional searches of this database were carried out using keywords from the original article title or the full journal title. This retrieved the retraction notices where the journal editor was cited as author and the title was given as ''Retraction notice.''

Annotation of retractions in online journals
By the second stage of the study, several further articles had been retracted (or the retraction date was unclear), coming to a total of twenty-four articles retracted from nine journals between April and December 2009. One article from each of the nine journals was chosen to assess how they were annotated as retracted: (1) the contents page of the relevant issue of the online journal was scanned; (2) the hypertext markup language (HTML) copy of the paper was viewed; and (3) the portable document format (PDF) version of the paper (where available) was also viewed. This article has been approved for the Medical Library Association's Independent Reading Program ,http://www.mlanet .org/education/irp/.. Table 3 and Table 4 are available with the online version of this journal.

Recording of retracted articles on bibliographic databases
Tables 1 and 2 (further information in Table 3, online only) provide details on the eighteen articles that had been retracted at the time of the searches, whether a record of the original article was identified on each database and whether there was a note that the article had been retracted.
MEDLINE performed best. Seventeen of the 18 records were found (94%), and all were clearly identified as having been retracted by an annotation of ''Retracted Publication'' at the end of the title.
Seventeen were found in EMBASE, one of which was identified as having been retracted (6%) by the inclusion of a separate record of the retraction notice. There was no indication for the remaining 16 records that the original article had been retracted.
Fifteen of the 18 original articles were identified on CENTRAL, 12 of which (80%) were annotated as retracted. There were 6 duplicate records. In 5 of the duplicate pairs, both records were annotated as having been retracted either by having a note added to the comments field or by having a notice of retraction added at the end of the title. There was 1 pair of duplicate records relating to the same paper, where 1 record indicated that the paper had been retracted, while the other record did not contain this information. Of the remaining 9 records, 6 contained a note of retraction and 3 had no indication that the paper had been retracted.

Annotation of retractions in online journals
The relevant online contents page for one retracted paper from each of the nine journals was searched. In four of the nine journals, the title on the contents page did not state that the article had been retracted (details in Table 4, online only). Where there was an HTML fullpaper version available, it was clear that the article had been retracted in all journals except one.
PDF versions of articles were available in six journals. When these were opened, there was no indication on three PDFs that they had been retracted. In one journal, the original paper had been deleted and was no longer listed on the contents page (Table  4, online only). Through the University of York library service, it was possible to freely access online the retraction notice in all journals except one.
In four of the nine journals, there was a risk of missing that a paper had been retracted. In one journal, there was no indication that the article had been retracted on the contents page or the PDF article; therefore, a user browsing the online journal who had clicked from the contents page to the PDF would not be alerted to the retraction. There was a notice of the retraction on the HTML version in this journal, though this had limitations as it was inserted at the bottom of the bibliography. Similarly, in the second journal, there was no indication on the contents page or the PDF version of the article that it had been retracted. The notice of retraction was only on the HTML version, though in this case, it was placed at the top of the page, below the title, where it could be easily seen. In the third journal, there was no link between the retraction notice and the original article on the contents page or HTML or PDF versions of the article. In the fourth journal, there was no link between the original paper and the retraction notice, and the original article had been removed from the online journal.

DISCUSSION
Of the three databases searched here, EMBASE was the least timely: The majority of articles had not been annotated as retracted three months to six months after formal retraction in a journal. In addition, identifying retracted records on EMBASE is potentially problematic as the original record remains on the database but is not annotated as retracted. The retraction notice is treated as a separate record, with the author usually cited as the editor issuing the retraction notice, and the notice is assigned the subject heading ''retracted article.'' As there is no direct link from the original article to the record for the retraction notice, it is easy to identify or retrieve the original record without noting the associated retraction.
In contrast, on MEDLINE, the original record for the paper that has been retracted remains in the database with a note about the corresponding retraction notice added after the title [10]. The record's publication type is also amended to include ''Retracted Publication.'' The retraction notice is also included in the database as a separate record with a publication type of ''Retraction of Publication.'' CENTRAL uses a similar approach, though there are also records where details of the retraction have been added to the comments (CO) field.   Table 4, online only.
{ Includes one partial statement.

Brief communications: Wright and McDaid
There was considerable variability amongst the online journals in how they displayed retracted information. This was surprising given that electronic journals can easily and reliably annotate retracted articles, compared, for example, to the situation with paper journals.
Some of the journals made the retractions very explicit. Best practice was when the retraction was clearly labeled in the title of the article on the contents page, there was a link to the retraction notice that was easily obtained, and ''RETRACTION'' had been embedded as a watermark in the PDF. There were other cases where users could have inadvertently downloaded the full paper and not know that it was retracted. For example, where the retraction was not noted on the contents page and a reader clicked directly on the full-paper PDF icon, but the retraction was not displayed there either. In some cases, the paper and retraction notice had not been linked, the retraction notice was positioned on the page where it could be easily overlooked, or it was not available. In one case, access to the recent online retraction was restricted by the terms of the subscription at the authors' institution. This limitation should be reconsidered by publishers so that free access to retractions for all users is enabled.

IMPLICATIONS
The findings of this research have implications for publishers and users of bibliographic databases and electronic journals. Retracted records should be quickly annotated, and the methods used should be clear and consistent. Although retraction of papers appears to be increasing [11], it is not a common occurrence [12], so even an experienced librarian or information specialist may not be familiar with how bibliographic databases deal with annotating records to indicate retraction.
For those undertaking searches, the MEDLINE database is the most helpful in terms of both speed of updating and clarity of method. However, even when using this database, there is a risk in screening records that a retraction notice can be missed, therefore caution should be exercised. For example, it should be standard practice in all systematic reviews to check the final list of included studies against the MEDLINE list of retracted articles [12].
The findings suggest a strong need for guidance to publishers. Yet such guidance is currently available. The International Committee of Medical Journal Editors recommends that, in addition to a retraction notice appearing on a numbered page in a prominent section of the online journal, it should be listed in the table of contents page and included in the title of the original article. They also state that an electronic journal should, under no circumstances, remove an article, as electronic articles should be stored in a permanent archive for historical record [13]. This survey of a small number of journals indicates that these recommendations are not followed consistently.
Similarly, the guidance from the Committee on Publication Ethics recommends that retracted articles should be clearly identified in all electronic resources: specifically that they (i) be labeled in such a way that they are identified by bibliographic databases, (ii) appear in all electronic searches for the publication, and (iii) are clearly annotated as retracted on the journal's own website [14].
However, not a great deal appears to have changed since Sox and Rennie made recommendations following the Poehlman case in 2005 [5]. There is more work to be done in encouraging journals to follow current guidelines. In the meantime, users of bibliographic databases and online journals should be aware that there is variability between the databases in timeliness of annotating retractions and between journals in the clarity of annotations.
A limitation of this study is that it is a small sample of all retracted records and uses a small sample of all health care journals. Only three bibliographic databases were assessed; however, they are the most commonly used in systematic reviews related to health care [8]. The small sample of records is less of a limitation in the evaluation of the bibliographic databases. Using the case study allowed assessment of a reasonable volume of retracted papers across a single timeframe. Further investigation of how online journals in health fields other than anesthesiology, and indeed beyond health care, deal with annotating articles as retracted would be beneficial. Scaling this case study up to a large random selection of retracted articles across disciplines would provide information on the scale of the problem.