Drilling deeper into the core: an analysis of journal evaluation methodologies used to create the ‘‘Basic List of Veterinary Medical Serials,’’ third edition

Objective: The paper analyzes the journal evaluation criteria used to create the third edition of a core list of veterinary serials to determine the impact of each criterion on the final composition of the list in order to assess the value of using multiple criteria in creating a core list. Methods: Three additional lists were generated from criteria that were previously combined to prepare the third edition of the ‘‘Basic List of Veterinary Medical Serials’’: a list based on journal recommendations from veterinary specialty organizations, another list based on journals selected by veterinary librarians, and a list based on both indexing coverage and scholarly rank. The top fifteen journals in each of the three lists were then compared to reveal potential biases. Subject representation on the full lists generated by each of these methods was also compared. Results: The list based on journal recommendations from veterinary specialty organizations exhibited a focus on clinically relevant titles. The list based on veterinary librarian recommendations resulted in the broadest subject coverage. The list based on indexing and scholarly rank, while emphasizing research titles, produced the largest number of unique titles. Conclusion: A combination approach that includes objective evaluation measures and practical input, whether from librarians or discipline experts, can improve coverage and can result in a list that balances research-based with clinical practice journals.


INTRODUCTION
Library professionals have created core lists of journals for nearly every discipline they serve. The major uses for core lists include guidance in journal selection, retention decisions, and collection assessment. Core lists have proved their value as collection management tools to the point that many existing lists continue to receive regular updates and new core lists are emerging to address the growth of interdisciplinary programs. Often core lists are relied on because it is presumed that professionals have ''already performed the background work by assessing journal quality, relevance, subject and patron need'' [1]. Recent studies of core lists have explored the methodologies used for title selection and have described a wide spectrum of possible approaches to creating a core list, ranging from strictly bibliometric measures to processes based primarily on subjective opinion [2][3][4]. This article explores the impact of different criteria used to build a core list in veterinary medicine and compares resulting lists generated from each of these individual criteria.
In 2008-2009, four different criteria were gathered and analyzed to create the third edition of the ''Basic List of Veterinary Medical Serials'' [5]. Coverage in abstracting and indexing databases (AI) was selected as an indicator of a journal's importance to the field, its quality, and the accessibility of its content. Journals that appeared in any of four AI databases-CAB Abstracts, Agricola, MEDLINE, and Web of Science-N Use of indexing or scholarly rank is often thought of as a limiting factor, but in this study, the journals that were scored solely on these two criteria resulted in the inclusion of the greatest number of unique titles. N Librarian input produced the broadest subject coverage of any list.

Implications
N An analysis of the strengths and weaknesses of various methodologies reveals clear evidence for using both objective and subjective criteria in developing a core list. Collection development decisions would equally benefit from this approach.
N Comparison of the top titles on core lists underscores the value of input from practitioners or discipline experts in collection decisions.
N Bias toward clinical content revealed in the input from the veterinary specialty organizations indicates the importance of understanding the purpose and compilation methods used in creating recommended lists before using them in collection development.
were deemed relevant to veterinary science and were weighted more favorably. The journal's ranking based on scholarly impact was another factor evaluated in considering titles for the ''Basic List.'' Thomson Reuters' Journal Citation Reports (JCR) was viewed as the leading measure of impact. With the previous two editions of the veterinary core list, JCR had been the only measure considered, but in an effort to balance potential drawbacks with using this single ranking, the SCImago journal rank (SJR) was added as a second scholarly impact measure [6]. Another quality measure added in creating the updated core list of veterinary serials was recommended reading lists compiled by veterinary specialty organizations recognized by the American Veterinary Medical Association (AVMA). The committee producing the ''Basic List'' reasoned that journals appearing on the recommended reading lists had been judged valuable from the perspective of practicing veterinarians. The final criterion considered was a survey conducted among veterinary librarian colleagues to solicit their opinions on journals essential to a core list. The committee felt confident that these criteria represented an ideal balance between objective bibliographic measures and subjective journal measures. They also believed that each of these components provided a specific journal quality measure and that a calculated mix of the criteria would produce a core list that was balanced between clinical and research titles and provided adequate coverage of important subject areas in veterinary medicine. The creation of such a list was the intended outcome of the committee's efforts. A scoring matrix was then devised to apply a weighting system to the four different facets, and each journal was given an overall score. A minimum score was established to determine the final ''Basic List'' of journals.
A key question that emerged during the previous effort was the extent to which the titles included in the ''Basic List'' would have changed if a different mix of selection criteria were applied. In an effort to validate the hypothesis that a combination of objective and subjective criteria collectively produced a core list with improved coverage with respect to the mix of research and clinical journals, a methodical analysis was conducted to determine the various lists that would have resulted if each of the above-mentioned selection criteria had been applied individually. Richards, a noted health sciences collection development expert, once asserted, ''by continually reviewing the factors that comprise the selection process, we can minimize the level of bias in our decisions'' [7].

METHODS
To compare the effect of different criterion on creation of the ''Basic List,'' 2 analyses were performed. First, 3 different journal lists were created by separating criteria previously gathered to prepare the third edition of the ''Basic List.'' Creating the ''Basic List'' involved assessment of 4 criteria to determine an overall score for 238 journal titles. Criteria were weighted as follows: indexing coverage, 4 points; scholarly impact, 8 points; inclusion on a veterinary specialty organization's recommended reading list, 12 points; and a rating from veterinary librarians, 16 points.
The first of the 3 new journal lists created in this way was based on objective bibliometric data, which included the AI coverage together with the scholarly rankings using the point value assigned in creating the original ''Basic List'' [5]. The second journal list was based on the scores recorded for journals appearing on recommended reading lists from veterinary specialty organizations (Table 1). Lastly, a third journal list was generated based only on scoring related to the librarian survey. The top 15 journals were selected from each of the 3 versions and compared against the top 15 titles from the 123 titles forming the ''Basic List'' (''Basic Top 15'') to detect overlap, as well as unique titles (Table 2). Analysis was also performed on the subject representation and number of titles within the various subject areas for each list in order to study differences and identify any potential biases ( Table 3). The decision to analyze only the top 15 titles was motivated by a desire to duplicate the limits frequently encountered in a real world collection development situation, even with the aid of a core list.
Second, full journal lists based on each individual criterion were analyzed. Each list had a different number of titles, reflecting the original method used to compile the ''Basic List.'' The librarian survey asked respondents to mark each of 238 titles as either key to a core list, not key but to include in a core list, no opinion, or remove from the list. The survey was not expressly designed for straight ranking of the titles but rather for assigning ratings to all the titles under consideration for the final ''Basic List.'' In the case of the AI and scholarly rank, nearly every one of the 238 titles was covered in at least one of the 4 targeted indexes, resulting in a list of 220 titles. The comprehensive nature of indexes such as CAB Abstracts earned some of the titles their only point. The full journal list based on specialty board recommended reading resulted in the selection of only 80 out of 238 journal titles.  The 123 titles on the full ''Basic List'' spanned 36 subjects, while the ''Basic Top 15'' were concentrated in 9 subjects. The subject categories appearing in the ''Basic Top 15'' were: equine medicine, food animal medicine, internal medicine, microbiology and immunology, pathology, research, small animal medicine, surgery and anesthesiology, and zoo and exotic animal medicine. The list derived from the librarian survey resulted in journals representing 12 subject categories, including the same 9 subjects appearing in the ''Basic Top 15,'' plus avian medicine, parasitology, and radiology. Subject representation resulting from the specialty organizations' recommended reading matched 8 of the same 9 subject categories represented in the ''Basic Top 15,'' but included 1 title in the subject category of emergency medicine and critical care, rather than a title representing microbiology and immunology. In contrast to the other lists, the list based on AI and scholarly rank covered the smallest number of subject categories, at 8, but was the only list with subject representation in public health and reproduction for titles in the top 15. Subject areas primarily considered clinical-such as food animal medicine, small animal medicine, surgery and anesthesiology, and zoo and exotic animal medicine-were not represented in the top 15 on this list but included on all other lists. Comparing the full lists generated by using each methodology separately (Tables 4-7) shows that the expanded list based on AI and scholarly rank included most of the clinical subject areas. This could be expected because (a) the list had no scoring cutoff point imposed and (b) a large number of titles are published in subject areas such as internal medicine (70), food animal medicine (22), and zoo and exotic animal medicine (16). A list based on specialty board recommended readings resulted in the selection of only 80 out of 238 journal titles and was weighted solidly in clinical medicine. Ten of the 36 subject categories in the ''Basic List'' were not covered at all, including areas such as biotechnology, business, education, history, and law. For all these individual criteria lists, the subject emphasis observed in the top 15 lists remained true, though it was less dramatic in the expanded versions, as eventually more subjects were included.

DISCUSSION
Literature on core lists has explored various methods for analyzing journal titles, but much less insight has been provided on the strengths and weaknesses of different approaches. Analysis of AI coverage and scholarly rank are bibliographic measures often considered to be important components in producing a core list. Coverage of particular journal titles in major AI databases is a well-known quality measure, because the processes for having a journal added to particular indexing databases demonstrates application of special quality standards. For example, editors and publishers seeking to have their journal indexed in MEDLINE must submit an application and provide a representative number of journal issues for committee review regarding ''quality, originality, and importance of the scientific content'' before indexing coverage is granted [8]. Web of Science uses a similar process of evaluating journal issues for possible coverage, assessing ''the journal's basic publishing standards, its editorial content, the international diversity of its authorship, and the citation data'' [9]. Also important to some indexing services is to be as comprehensive as possible on a subject. CAB Abstracts aims to ''process all relevant publications, including less well-known and non-English journals and those published by independent and learned publishers,'' though still to apply quality standards by reviewing journals ''on the basis of subject matter, potential yield and geographic origin'' [10].
In terms of scholarly rank, numerous studies of JCR impact factors have acknowledged its authoritativeness, reputation, and wide use among scholars, but many have also noted its limitations. Some of the problems associated with the impact factor calculation include time lag, journal self-citations, bias toward English-language journals, types of articles published (i.e., the effect that review articles have on increasing a journal's score), variations based on research field, and journal history and format [11,12]. Studies have also noted drawbacks with the counted citations, for example, ''citations are not always an indicator of quality but may be cited to point out inaccuracies or errors in the research'' or ''the extent to which the author actually used the cited document'' [11,12]. Because impact factors are calculated each year, one library scholar advises ''to review a journal's ranking for the last several years when reaching journal collection management decisions'' [11]. A 1979 study comparing core lists of medical journals discussed the ability of JCR data to measure a journal's importance but concluded that it ''be used only in conjunction with other collection development tools'' [13].
Using the combination of these two measures (AI and scholarly rank) as the sole metric for compiling a core list could be problematic, because such a calculation does not consider the importance of particular journal titles based on patron use or value to professional practice. However, compared to the lists based on other criteria, the top fifteen journals from the list based on AI and scholarly rank produced the greatest number of unique journal titles. Unsurprisingly, the top fifteen titles based on AI and scholarly rank favored journals with a research emphasis, due to the citation frequency of these types of journals and their resulting higher scholarly rank.

Ugaz
Subject categories represented by the unique titles, and throughout the AI and scholarly rank core list in general, had a solid research focus (Table 3). Given the decreased emphasis on clinical titles when using this approach for creating a core list of veterinary journals, would the same be true for core lists in other disciplines? The subject representation achieved in this list based on AI and scholarly rank, with a strong emphasis on journals with a research focus, supports the benefits achieved by complementing this metric with input from librarians and clinical practitioners. For the full third edition of the ''Basic List,'' reading lists from veterinary specialty organizations were consulted to identify journals that are deemed key in preparing for a specialty board exam and that are titles libraries would feel obligated to include in their collections. Although the use of standard lists in collection development might be considered an objective measure in the evaluation of a resource, recommendations based on the judgment of practitioners in a professional organization are similar to the subjective input typically gathered from faculty as a factor in collection decisions. Examples of journal lists that have been prepared by other professional organizations include the American Association of Colleges of Pharmacy (AACP) and the American College of Physicians. The intention of the endorsed list is sometimes reflected in the approach taken to generate the recommended resource list. The ''AACP Core List of Journals for Libraries that Serve Schools and Colleges of Pharmacy'' is intended as a collection development tool for libraries supporting pharmacy programs and is prepared by a committee of the AACP. The introduction to the list indicates that indexing coverage and a vote from section members are significant considerations, but it also states that editors and contributors to the list have past or current library experience and provide ''recommendations for titles to be included in pharmacy library collections'' [14]. Though it ceased in 1997, ''A Library for the Internists'' was a list of recommended resources ''designed to help internists and community libraries acquire and apply current knowledge in internal medicine'' and was assembled with input from medical information experts, but the chief contributors were practicing general internists [15]. The resource lists that the veterinary specialty organizations provided often noted that the books and journals listed were recommended reading in  preparation for a board certification exam and, in some cases, were the sources of exam questions. In describing their suggested reading materials, the American College of Theriogenologists (ACT) stated, ''references in bold are considered to contain relevant contributions to the field of theriogenology and may be helpful in the candidate's preparation for completion of the ACT certifying examination'' [16]. In terms of subject representation achieved, the veterinary specialty organization core list matched very closely with subjects included in the ''Basic Top 15,'' the exception being one title in the subject category of emergency medicine and critical care in place of a title representing microbiology and immunology. One explanation for the substitution lies with the veterinary specialty organizations currently formed and recognized by the AVMA and for which reading lists were successfully obtained. The subject categories associated with journals that ranked in the top fifteen are a direct reflection of the specialty organizations available for consultation (e.g., American Board of Veterinary Practitioners, American College of Veterinary Internal Medicine, American College of Veterinary Emergency and Critical Care, and American College of Zoological Medicine) (Table 1). As research journals were emphasized in the core list based on AI and scholarly rank, internal medicine and other clinical medicine journals tended to be favored in the list based predominantly on input from clinical practitioners ( Table 3). The influence of each criterion on the makeup of the ''Basic List'' and the balance between research and clinical journals that resulted when combining them becomes evident. This effect on the ''Basic List'' further supports an approach that includes practitioner input when assembling a core list for libraries serving users who are preparing for or working in the clinical setting.
The core list based on the librarian survey achieved the greatest subject representation at 12, more than either the ''Basic Top 15'' or the specialty organization core list. Three subject categories (internal medicine, equine medicine, and small animal medicine) were each represented by 2 titles and the other 9 subject categories each had 1 title (Table 3). While the subject emphasis seemed to lean slightly toward clinical medicine, the top 15 also had a mix of research journals. Besides highlighting the inherently broader perspective toward collection development, the librarian survey list reflects the value of consulting colleagues, reviewing the holdings of peer libraries, and consulting other sources of experiential input. The survey component used in creating the full ''Basic List'' asked veterinary librarians to specify which of   Ugaz 238 journals were essential to a collection. This input drew upon their knowledge and experience to identify titles demonstrating significant history of use, high quality, frequent patron requests, frequent appearance in search results, and high demand in interlibrary loan requests. It is acknowledged that survey responses are necessarily biased by the perspectives of the individual librarians at institutions with variations in research focus, curriculum emphasis, and curriculum design. In the case of the veterinary core list, survey responses collected from 24 librarians at different institutions showed collective agreement on most of the journal selections. Analysis of the full journal lists (expanded beyond the top 15) in essence supports the findings observed with the smaller subset of the top 15 for each list. The list based on the librarian survey results still had the broadest subject coverage; the full journal list based on AI and scholarly rank included more journals in research subject areas; and the list based on specialty board recommended readings favored clinical medicine titles. The subject emphases are less obvious in the full journal lists only because no cutoff or threshold was applied to limit the number of journal titles for each list, whereas a minimum score was used in the creation of the veterinary core list. Subject emphasis on research or clinical practice in the full lists is best observed by examining the number of journal titles within subjects and the percent of subject representation for each of the lists (Tables 5-7).

LIMITATIONS
Several changes in the list ranking veterinary titles in the SJR system have occurred since it was originally accessed in 2008. Journal titles that were previously assigned to the veterinary subject category are now in another subject area, and other journal titles that had not been in the veterinary subject area have been added. The current number of journals ranked in SJR for the subject of veterinary is 168 titles, whereas 151 titles were included when the SJR was originally accessed in 2008. It is not clear why the lists for that particular year did not remain static, but for consistency in this project, the data originally obtained in 2008 was referenced. Despite the relatively minor issues caused by the updates to the SJR list, it was still viewed as an enhancement to the ranking provided by the JCR list.

CONCLUSION
The process of analyzing the methodology used to create the ''Basic List'' presented the opportunity not only to validate the combination of criteria applied, but also to observe the influence exerted by each individual criterion on the composition of the list. Objective bibliometric measures such as indexing coverage and impact factor formed the foundation of the first and second editions of the ''Basic List,'' with a committee ultimately deciding on the final list of titles. In updating the third edition, the committee added new subjective measures in the form of input from a larger group of subject librarians and veterinary practitioners. This study tested the assumption that the selection criteria enhanced the third edition of the ''Basic List'' by examining the strengths and weaknesses of four different evaluation methodologies.
The inclusion of bibliographic measures strengthened the list by emphasizing key veterinary research journals, particularly in the subject areas of parasitology, microbiology and immunology, public health, and reproduction. Consulting journal recommendations issued by veterinary specialty organizations highlighted a bias toward clinically relevant literature. The librarian input produced the broadest subject representation, resulting in equal mix of research and clinical practice journals and suggesting an approach that considered multiple journal factors along with user needs. Further research could be done to explore whether this same effect would be seen in other disciplines. This analysis of the biases inherent with journal evaluation criteria supports a multicriteria analysis for the development of core lists, in particular the addition of input from librarian and discipline experts, to ensure balance in the distribution of titles and subject representation.
This analysis of the methodology supporting the collection development decisions involved in creating a core list has broader implications for collection development processes in general: the need to include input from a variety of sources, including both objective and subjective input; the value of soliciting input from discipline experts or practitioners; and the value of consulting librarian colleagues.