Do local citation patterns support use of the impact factor for collection development?

The Impact Factor and Collection Development 
The journal impact factor (IF) is reported by ISI in Journal Citation Reports (JCR). “A journal's impact factor is based on 2 elements: the numerator, which is the number of citations in the current year to any items published in a journal in the previous 2 years, and the denominator, which is the number of substantive articles (source items) published in the same 2 years” [1]. 
 
Much research has been done on IF as a measure of local journal use for collection development. Blecic used a comparison of three methods: in-house use, circulation, and citation to determine journal use, finding a “significant correlation” between the three methods and arguing that because of this correlation, only one type of data was necessary to make retention decisions [2]. MacDonald's study of online journal usage in relation to citation analysis examined whether online journal use (as measured by an academic library) and a library's publisher-reported full-text downloads predicted citations, finding that “citation is clearly related to usage” [3]. Using Biosis Previews, Davis identified core journals in the life sciences by analyzing the journals in which Cornell University authors published [4]. Davis concluded that the “generic metrics of the JCR simply cannot provide the campus-level data crucial to making informed decisions about the local importance of individual titles,” as argued earlier by Pan [5] and Chrzastowski [6].


The Impact Factor and Collection Development
The journal impact factor (IF) is reported by the Institute for Scientific Information in Journal Citation Reports (JCR)."A journal's impact factor is based on 2 elements: the numerator, which is the number of citations in the current year to any items published in a journal in the previous 2 years, and the denominator, which is the number of substantive articles (source items) published in the same 2 years."[1] Much research has been done on IF as a measure of local journal use for collection development.Blecic [2] used a comparison of three methods: in-house use, circulation, and citation to determine journal use, finding a "significant correlation" between the three methods and arguing that because of this correlation, only one type of data is necessary to make retention decisions.MacDonald's study of online journal usage in relation to citation analysis examined whether online journal use (as measured by an academic library) and a library's publisherreported full text downloads predict citations; the study found that "citation is clearly related to usage" [3].Using Biosis Previews, Davis identified core journals in the life sciences by analyzing the journals in which Cornell University authors published [4].Davis concluded that the "…generic metrics of the JCR simply cannot provide the campus-level data crucial to making informed decisions about the local importance of individual titles," as argued earlier by Pan [5] and Chrzastowski [6].

Collection development at the Indiana University School of Medicine
The Indiana University School of Medicine (IUSM) has nine campuses across the state and is the second largest US medical school.IUSM expects library collection development to support funded research.Traditionally, the IUSM Library's collection development policy has been based on an analysis of online and print cost/use ratios [7,8], IF, and interlibrary loan requests.The purpose of this study was two-fold: to determine whether a journal's IF rank reflects its citation frequency among local researchers, and to evaluate whether current methods of journal selection and deselection are meeting the needs of IUSM-sponsored researchers.
The authors approached the problem from a unique perspective: analyzing references in published articles by funded researchers in the small but highly grant-productive Psychiatry department in the IUSM.The authors hypothesized that given the specialized nature of research conducted by grant recipients, the journals they cited might not be identified by a high IF.
Psychiatry was ranked fourth in grants awarded out of 26 IUSM departments, and considering its size (70 full-time faculty), the department's efforts have been significant.The 65 psychiatry grants active during the study period of Fiscal Year 2004/2005 are delineated by subject area in Table 1.Grants in the areas of alcoholism, schizophrenia and Alzheimer's disease accounted for half of the grants both in terms of quantity and dollars awarded.The department's output provided a manageable data set to investigate the citation patterns of the grant recipients, which could then be compared with the IUSM Library's local holdings and the IF of the journals cited.

Study Sample
Principal investigators (n=22) on grants active in fiscal year 2004/2005 (the most current year available) from the IUSM Department of Psychiatry were identified by the university's Department of Research and Sponsored Programs.Searches for publications of faculty members identified were conducted by librarians experienced in using Thomson Scientific's Web of Science (WOS), which includes Science Citation Index, Social Science Citation Index and Arts & Humanities Citation Index.To ensure comprehensive search results, each psychiatry faculty member was identified by research interests, alternate names, previous affiliations, and any other useful information which could be obtained from the IUSM Website.Articles published by faculty members when they were at other institutions were included.The searches were limited to publication years 2000-2004, the period immediately preceding the active grant report, and included all publication types indexed in the database.All articles by an author were included regardless of author list position (e.g.first author, last author, etc.).

Data Collection
The complete bibliographic record and cited references for each publication were saved as a tabdelimited file and imported into a Microsoft Access database table.Since some publications were authored by multiple faculty members in the study, records for duplicate publications were removed from the table.After removing duplicates, 429 unique publications by the 22 faculty members remained.Out of those 429 publications, 405 had cited references (CR).The CR field from each bibliographic record was then stripped and put into one table.The five main data elements from each cited reference (Author, Date, Source, Volume and First Page) were parsed into separate fields in the table to facilitate analysis.

Data Analysis
Data analysis focused on the 77 titles in the Psychiatry subject category of JCR for all years from 2000-2004.This list of titles along with impact factors for 2000-2004 was imported into the Access database.To address the year-to-year fluctuation in IF identified by Altmann [9], an average IF for the five-year period was calculated.Although Nisonger [10] has identified a bias in using average IF, he concluded that for practicing librarians the degree of inaccuracy likely does not justify the effort to correct it.
A database query was written to match the journal titles in the CR with the journal titles in the JCR Psychiatry list.The titles were then ranked by number of CR and the five-year average IF.The ranks of titles with equal numbers of CR were averaged.For example, Australian and New Zealand Journal of Psychiatry and Schizophrenia Bulletin were both cited ten times, ranking them 36 th and 37 th .Since it would not be accurate to rank one higher than the other, the ranks were both averaged to 36.5.
After ranking all of the titles, a rank difference was calculated by subtracting the IF rank from the rank by number of CR.The final data element added was an indication of whether the IUSM Library subscribed to the title.Only titles subscribed to for the entire five-year period were counted to maintain consistency across all titles.The results are shown in Table 2 ranked by number of CR.

Impact Factor Compared to IUSM Psychiatry Faculty Cited References
The authors found a strong positive rank correlation between IF and number of CR as shown by the Spearman rank correlation coefficient of r = 0.788 (Figure 1).The difference in rank ranged from 0-41.5 with an average difference of 11.16 and median of 3.75.Only fifteen titles had a difference in rank greater than twenty (Table 3).Nine of those titles ranked higher by IF and six ranked higher by number of CR.
The average IF for JCR Psychiatry category journals not cited by IUSM Psychiatry researchers in their publications was 0.8121 (range=0.2412to 1.7198).These journals were all in the bottom 50% of the JCR Psychiatry category as ranked by IF.The average IF for JCR Psychiatry journals cited by IUSM researchers was 2.6221 (range=0.5966to 11.4214).

IUSM Library Holdings compared to IUSM Psychiatry Cited References
The IUSM Libraries subscribed to 47 (61%) of the 77 JCR Psychiatry category journals.Ninetyfive percent (n=3,083/3,232) of the CR in IUSM Psychiatry faculty publications were titles to which the library subscribed.Of the 77 titles in the JCR Psychiatry category, 61 (79%) were cited by the IUSM Psychiatry Faculty.Of those 61 titles, seventeen (28%) were not current library subscriptions during the study period.The average IF for these seventeen titles was 1.8832 (range=0.358to 3.7134).Eight of the seventeen titles not held ranked in the top 50% by IF, and three ranked in the top 50% by CR.Only one of the seventeen titles not subscribed to, Schizophrenia Research, was in the top 50% in both rankings.
(n=149) of cited references were to journal titles that were not current library subscriptions.
While none of those seventeen titles are freely accessible, seven of them became accessible at IUSM during the study period through consortial agreements.It is impossible to say whether this finding shows that researchers only cite the journals available to them in the library or that the library's collection supports the research of the Psychiatry department well.However, since this study included a small number of publications by IUSM Psychiatry faculty when they may have been at other institutions (18%; n=79/429), their cited references were not completely based on the collections of the IUSM libraries.

Limitations
Since this study relied exclusively on WOS, any publications by the authors in the study that were not indexed in that database were excluded from the study.The authors studied were also from only one department in the IUSM, so the results may not be generally applicable to all IUSM researchers or to researchers at other institutions.In addition, impact factors can only be compared within a journal category (in this case, psychiatry).Thus inferences drawn from the data are necessarily limited to the area under study.

CONCLUSIONS
This limited, "proof of concept" study suggests that IF is a fairly accurate reflection of local psychiatry research emphases as measured by the cited references in the publications of local authors.The results also provide evidence that the IUSM Libraries collection is meeting up to 95% of the research needs of IUSM's psychiatry researchers.Since IF is not used in isolation, and the process of local citation analysis is quite time-consuming, the authors recommend the