|
|
J Med Libr Assoc. 2008 January; 96(1): 63–66. doi: 10.3163/1536-5050.96.1.63. | PMCID: PMC2212341 |
Analyzing the impact of an author's publications Lee A. Vucovich, MLS, AHIP, Jason Blaine Baker, MLS, and Jack T. Smith, Jr., MLS Lee A. Vucovich, MLS, AHIP, Email: lvucovi/at/uab.edu, Instructor and Assistant Director for Reference Services; Jason Blaine Baker, MLS, Email: jbb42/at/uab.edu, Instructor and Reference Librarian; Jack T. Smith Jr., MLS, Email: jsmith/at/uab.edu, Library Associate; Lister Hill Library of the Health Sciences, University of Alabama at Birmingham, 1530 Third Avenue South, Birmingham, AL 35294-0013 Received June 2007; Accepted September 2007. Readers may use articles without permission of copyright owners, as long as the author and MLA are acknowledged and the use is educational and not for profit. Information professionals are called on to determine how best to measure the impact of an author's articles, and citation counts are often regarded as one method for obtaining a quantitative expression of the utilization and contribution of a particular published paper. As Meho states, citation analysis assumes that influential works or scientists are cited more often than others [ 1]. Egghe and Rousseau claim that citation counts are based on four important assumptions: an article's citation implies use of that document by the citing author; the citation reflects the merit (quality, significance, impact) of the article; the references are from the best possible works on the topic; and the cited articles are related in content to the one in which they are used [ 2]. Traditionally, the peer-review process has been used to assess article quality. Currently, there is a global trend toward the development, refinement, and increased use of quantitative metrics, particularly those resulting in “quantifiable, post publication quality assessment” [ 1, 3, 4]. However, determining impact by citation analysis can be controversial; in some cases, works are cited to point out errors and inaccuracies in the research. Additionally, long articles are often cited more frequently, and some reference lists contain erroneous citations, which can skew results. Finally, journal visibility and prestige affects dissemination, and self-citation can artificially inflate citation counts [ 1, 3, 5– 8]. Despite these concerns, citation analysis remains a useful tool for assessing faculty research publication. The journal impact factor (JIF) was developed to facilitate comparison between citation rates of journals and evolved as a measurement of journal quality on the assumption that a higher citation rate equaled a higher quality journal [ 9]. This assumption causes concerns, as Amin and Mabe indicated, because it is often used as the “chief quantitative measure of the quality of a journal, its research papers, and the researchers who wrote the paper” [ 10]. Many authors have noted other factors that affect the actual impact factor number: (1) research field, (2) type of journal, (3) average number of authors per paper, (4) size of the journal, and (5) two-year measurement window. Other limitations are that JIFs are biased toward US publishers, a small percentage of articles is highly cited, and the JIF may be easily manipulated [ 1, 3, 10– 12]. Also of note is the fact that a journal may not yet be indexed in Web of Science (WOS) or tracked in the Journal Citation Reports (JCR) database long enough to have an impact factor. For these reasons, many have cautioned against using JIF to judge the quality or impact of individual papers or authors [ 9, 13]. Vieira and Faraino, however, used JCR to analyze the research record of their institution's list of faculty publications [ 14]. They pointed out that JCR can be an important research tool in indicating how faculty authors were citing the literature. More recently, Saha et al. found a strong correlation between the quality ratings of surveyed physicians of nine general medicine journals and their impact factors [ 15], while Yue et al. found that clinical and research neurologists' ratings of journal quality also correlated with impact factors [ 16]. Rice et al. provided critical information about the statistical formulas used to calculate the reliability and validity of citation data [ 17]. THE UNIVERSITY OF ALABAMA AT BIRMINGHAM EXPERIENCE In October 2006, the Reference Department of the Lister Hill Library (LHL) of the Health Sciences at the University of Alabama at Birmingham (UAB) received a request from a university administrator to ascertain which papers or journal articles written by several UAB authors over the past ten years have had the greatest impact. The administrator made no distinction between research articles or other article types. To fulfill this request, WOS searches for each different author were performed. The same search strategy in WOS was used for each author. The requestor and the librarians mutually agreed that the search would utilize the author's last name with first and middle initials. To address issues of locale, the city “Birmingham” was used in the city (CI) field instead of zip codes. The CI field was included in the search strategy as the administrator was only interested in the publications that the authors had written while affiliated with UAB. Using the CI field also helped eliminate authors at other institutions with similar last name and initial combinations. The librarian informed the requestor of the various limitations of this search methodology: that the articles must all be signed in the same naming convention and that the city “Birmingham” may be located outside Alabama. Due to the uniqueness of the authors' names coupled with the city, false drops were not expected. The search was limited to the years of 1995 to 2006, and results were then sorted by the number of times cited. The librarians then used the WOS Results Analysis feature to obtain a report showing the title of the journal and the number of articles by the author being searched that were published in that journal. Results were sorted by record count with the minimum record count set to one. The librarians then utilized JCR's journal summary feature to sort the journals in specific subjects by their impact factors. For this particular request, the source title list was reviewed and the librarians identified the major categories (e.g., surgery, internal medicine) using the subject categories identified in the JCR record for each journal. A list of impact factors for journal titles in the appropriate areas was generated and included in the packet that the librarians hand-delivered to the requestor. A distinction was made by subject to provide a more representative comparison given variations of impact factors in subject categories. The requestor's packet included the following items: WOS author search sorted by times cited, WOS results analysis with the records ranked by record count, JCR subject category list, and the journal summary list for each subject category, sorted by impact factor. The packet also included a cover letter describing the search process and explaining that determining the impact of an author's work requires caution. The variables to be considered included: (1) number of times the journal article has been cited (are self citations included?); (2) author's position in the author string (if the article is the product of an author's lab, the author will usually be listed last); (3) impact factor of the journal (viewed generally or within its subject categories); (4) date of publication (more recent articles may not have been published long enough to have been cited numerous times); and (5) subject area of the journal (determining if this is a large subject area in terms of the number of journals published in that subject or a really narrow specialty). The librarians informed the requestor that, given the information provided, it was the requestor's responsibility to analyze the data and determine the appropriate value or weight to give to each piece of information. Reference departments in other medical centers may often struggle with similar requests. Published literature indicates that various approaches and tools are available for assessing the impact and quality of a researcher's work. While the LHL librarians decided to utilize JCR and WOS, the emergence of additional web-based citation analysis tools has had an impact on citation analysis and provides a number of new quantitative measures to be considered. In 2004, two primary competitors to WOS became available: Elsevier's Scopus and the freely available Google Scholar (GS). Several groups have compared these databases and have concluded that each of the three databases returned unique material [ 1, 5, 8, 18, 19]. Scopus includes a larger number of international and open access journals than WOS, thus providing complementary coverage. Although GS has limited search features, it includes other unique items such as book chapters, dissertations, electronic prints, and research reports. For 25 highly cited authors in the field of information science, a comparison of WOS, Scopus, and GS found that Scopus and GS increase the citation counts by 35% and 160%, respectively, revealing the importance of using several citation sources to judge the true impact of a scientist's work [ 1]. Jasco compared citations to a single paper (Science 1955;122:108–11) for the 1996–2005 time period [ 19]. Although WOS, GS, and Scopus returned a similar number of records, only 33 citing papers were common in the 3 result sets, leading to the conclusion that “a single database cannot provide comprehensive citation coverage.” In addition, the various databases offer different strengths as administrative tools and provide alternative ways to analyze the data [ 8]. Other web-based tools provide different approaches for measuring quality or impact. Introduced in 2001, the subscription-based Faculty of 1,000 offers a peer-reviewed alternative to citation analysis. Each month, over 1,000 experts select 2–4 papers in the biomedical fields and provide comments and grades for all [ 3]. An editorial in Nature Neuroscience noted a study that suggested that this tool provides an excellent correlation with JIF in the field of neuroscience [ 20]. Measuring the number of times an article is downloaded is also under discussion as a measurement tool or analysis method [ 21]. Dong et al. contend that online availability increases JIF in a positive manner [ 6]. Meho noted strong and positive correlation between download counts, citation counts, and JIF [ 1]. More research comparing measurement tools and the impact of downloaded articles is needed. Two interesting new approaches to citation analysis are PubFocus and h-index. PubFocus [ 22] is a web service that performs statistical analysis of the MEDLINE/PubMed search queries, enriched with the additional information gathered from journal ranking, and that incorporates the number of forward citations taken from PubMed Central or Google Scholar. The algorithm prioritizes citations and evaluates an author's impact on a field [ 23]. The h-index, proposed by Hirsch, is used to measure the impact of a scientist's body of work. The h-index correlates positively with citation counts, impact factors, publication counts, and peer evaluation of research impact and quality [ 24]. Currently, in WOS, the h-index is included in the Citation Report available with an author search, and it can easily be determined by using the Citation Tracker feature with an individual author search in Scopus. An analysis of both the quality and impact of an author's contribution requires a complete knowledge of the context of the request to determine the best approach to use, as too much is at stake if the process is oversimplified. Though this analysis focused exclusively on WOS, there are more tools that need to be further explored. Thompson Scientific's JCR and its JIF are still important tools that researchers will readily understand; however, use of these traditional tools introduces limitations in use and interpretation. Given the availability of multiple tools that may be considered in addition to JCR's citation analysis, such as GS and Scopus, it is up to librarians to carefully explain to researchers what tools are available, what criteria are used, and how the various pieces of this puzzle are put together to reach an answer that has both merit and validity. Further research is needed to determine if these emerging citation analysis tools will be able to withstand the rigorous testing and analysis to which WOS and JCR have been subjected. While there is great demand for easy quantitative methods to determine salary raises, tenure, promotion, and hiring, the experience of these reference librarians demonstrates that information professionals and librarians alike have a significant role in educating faculty and administrators about relying too heavily on one specific instrument or approach when making these decisions. The authors gratefully acknowledge the contributions of Nicole Mitchell, reference librarian, UAB Lister Hill Library of the Health Sciences, in helping finalize the manuscript for publication. - Meho LI. The rise and rise of citation analysis. Phys World. 2007. Jan; 20(1):32.–6.
- Egghe L, Rousseau R. Introduction to infometrics: quantitative methods in library, documentation, and information science. Amsterdam, The Netherlands: Elsevier Science Publishers, 1990.
- Cartwright VA, McGhee CN. Ophthalmology and vision science research. part 1: understanding and using journal impact factors and citation indices. J Cataract Refract Surg. 2005. Oct; 31(10):1999.–2007. [PubMed]
- Cheek J, Garnham B, and Quan J. What's in a number? issues in providing evidence of impact and quality of research(ers). Qual Health Res. 2006. Mar; 16(3):423.–35. [PubMed]
- Bakkalbasi N, Bauer K, Glover J, and Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006. Jun 29; 3(1):7.. [PubMed]
- Dong P, Loh M, and Mondry A. The “impact factor” revisited. Biomed Digit Libr. 2005. Dec 5; 2:7.. [PubMed]
- Nieminen P, Carpenter J, Rucker G, and Schumacher M. The relationship between quality of research and citation frequency. BMC Med Res Methodol. 2006. Sep 1; 6:42.. [PubMed]
- Thibodeau PL, Carden VRM. Citation analysis: innovations in how institutions view their publications [web document]. Durham, NC: Duke University, 2006. [cited 6 Jun 2007] <http://www.mclibrary.duke.edu/dukemac06/citationanalysisposter.pdf>.
- Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg Am. 2003. Dec; 85-A. (12):2449.–254. [PubMed]
- Amin M, Mabe M. Impact factors use and abuse [web document]. Perspectives Publish [serial online]. 2000;(1):1–6. [cited 6 June 2007] <http://www.elsevier.com/framework_editors/pdfs/Perspectives1.pdf>.
- Garfield E. The agony and the ecstasy—the history and meaning of the journal impact factor. Presented at: 5th International Congress on Peer Review and Biomedical Publication; Chicago, IL; Sep 16, 2005. [cited 6 Jun 2007] <http://www.garfield.library.upenn.edu/papers/jifchicago2005.pdf>.
- Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997. Feb 15; 314(7079):498.–502. [PubMed]
- Opthof T. Sense and nonsense about the impact factor. Cardiovasc Res. 1997. Jan; 33(1):1.–7. [PubMed]
- Vieira D, Faraino R. Analyzing the research record of an institution's list of faculty publications. Bull Med Libr Assoc. 1997. Apr; 85(2):154.–7. [PubMed]
- Saha S, Saint S, and Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc. 2003. Jan; 91(1):42.–6. [PubMed]
- Yue W, Wilson CS, and Boller F. Peer assessment of journal quality in clinical neurology. J Med Libr Assoc. 2007. Jan; 95(1):70.–6. [PubMed]
- Rice RE, Borgman CL, Bednarski D, and Hart PJ. Journal-to-journal citation data—issues of validity and reliability. Scientometrics. 1989. Mar; 15(3–4):257.–82.
- Burnham JF. Scopus database: a review. Biomed Digit Libr. 2006. Mar 8; 3:1.. [PubMed]
- Jacso P. As we may search—comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Curr Sci. 2005. Nov 10; 89(9):1537.–47.
- Revolutionizing peer review? Nat Neurosci. 2005. Apr; 8(4):397.. [PubMed]
- Coats AJ. Top of the charts: download versus citations in the International Journal of Cardiology. Int J Cardiol. 2005. Nov 2; 105(2):123.–5. [PubMed]
- PubFocus [web document]. PubFocus, 2006. [cited 21 Aug 2007]. <http://www.pubfocus.com>.
- Plikus MV, Zhang Z, and Chuong CM. PubFocus: semantic MEDLINE/PubMed citations analytics through integration of controlled biomedical dictionaries and ranking algorithm. BMC Bioinformatics. 2006. Oct 2; 7(1):424.. [PubMed]
- Ball P. Index aims for fair ranking of scientists. Nature. 2005. Aug 18; 436(7053):900.. [PubMed]
|
