• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jmlaJournal informationSubscribeSubmissions on the Publisher web siteCurrent issue of JMLA in PMCAlso see BMLA journal in PMC
J Med Libr Assoc. Apr 2009; 97(2): 93–101.
PMCID: PMC2670205

Development of a new academic digital library: a study of usage data of a core medical electronic journal collectionAn external file that holds a picture, illustration, etc.
Object name is mlab-irp.jpg

Barbara S. Shearer, MSLS, AHIP
Barbara S. Shearer, Director, Charlotte Edwards Maguire Medical Library, College of Medicine, Florida State University, Tallahassee, FL 32306-4300; ude.usf.dem@reraehS.arabraB;
Carolyn Klatt, MLIS*
Carolyn Klatt, Reference and Electronic Resources Librarian, Health Sciences Library, Memorial University Medical Center, Savannah, GA 31404 ; ude.recrem@ac_ttalK;
Suzanne P. Nagy, MSLS, SLIS, AHIP



The current study evaluates the results of a previously reported method for creating a core medical electronic journal collection for a new medical school library, validates the core collection created specifically to meet the needs of the new school, and identifies strategies for making cost-effective e-journal selection decisions.


Usage data were extracted for four e-journal packages (Blackwell-Synergy, Cell Press, Lippincott Williams & Wilkins, and ScienceDirect). Usage was correlated with weighted point values assigned to a core list of journal titles, and each package was evaluated for relevancy and cost-effectiveness to the Florida State University College of Medicine (FSU COM) population.


The results indicated that the development of the core list was a valid method for creating a new twenty-first century, community-based medical school library. Thirty-seven journals are identified for addition to the FSU COM core list based on use by the COM, and areas of overlapping research interests between the university and the COM are identified based on use of specific journals by each population.


The collection development approach that evolved at the FSU COM library was useful during the initial stages of identifying and evaluating journal selections and in assessing the relative value of a particular journal package for the FSU COM after the school was established.


  • The Florida State University College of Medicine (FSU COM) core list, created for developing an electronic journal collection to support a new medical school, is evaluated.
  • A moderate correlation is found between appearance on the core list and electronic journal use by the college.
  • Each of the four journal packages studied employs different pricing models. Each is evaluated for relevancy to the COM and to the larger university, and suggestions are offered for making cost-effective decisions.
  • Librarians at FSU are able to track both COM use and university use separately, and they can use data from this study to understand more about the amount of use each community makes of electronic resources and can identify areas of common interest.


  • A method is identified that will aid new academic medical libraries and libraries transitioning to digital collections in developing collections and evaluating e-journals and e-journal packages for relevancy to clinical medicine.
  • Data extracted on use of digital libraries provide an opportunity to make informed, cost-effective decisions not possible prior to the Internet.
  • Setting up unique EZProxy applications serving different populations is useful for learning more about use patterns of medical libraries that function in larger university environments.


The Florida State University (FSU) Charlotte Edwards Maguire Medical Library (MML), the first academic medical library created after the establishment of the web, was developed as a digital library at start-up in 2001. The journal collection development process was not influenced by ownership of a large legacy print collection, but rather concentrated on identifying key electronic journals of immediate value to an emerging medical school. A recent article reviewing the migration from print to electronic journal access as a case study at the University of Miami Louis Calder Memorial Library reported dramatic electronic journal usage increases from 1999 to 2003 and an 88% decrease in print journal use from 1995 to 2004 [1]. The creation of a core list specifically constructed to serve the needs of a large clinical clientele, primarily located off-campus and requiring online access, was seen as an essential step in meeting accreditation standards for the medical school, both during the start-up phase and later when it became established.

The MML supports a community-based medical school. Its primary clientele of authorized users includes 1,400 part-time clerkship faculty, 150 full-time faculty in medicine and the biomedical sciences, 440 medical students and biomedical graduate students, and 30 residents. With the exception of residents in 1 hospital, the library does not provide access to its collections to personnel at any of the affiliated hospital sites. However, because the MML is part of a larger university system, it provides access to its licensed e-journals to the 44,000 faculty and students in all areas of study at FSU. The University Library of FSU supports nursing, psychology, human sciences, biology, and other life sciences through individual or package subscriptions and consortium arrangements negotiated with most major scientific and biomedical publishers and through purchase of a sizable print collection housed separately in the FSU Dirac Science Library. All of these collections are available also to members of the FSU College of Medicine (COM) community and provide strong foundation collections in both clinical and biomedical research areas. It should be noted that the FSU COM has a growing biomedical research agenda that is supported by both the MML and FSU University Library collections. The FSU COM is somewhat unique in another way: the college includes a department of medical humanities and social sciences charged with integrating bio-psycho-social knowledge into the curriculum and the research agenda of the college.

A number of core lists and selection tools for medical and hospital libraries can be used for selection, retention, and evaluation purposes. These include the most widely used list, the “Brandon/Hill Selected List of Print Books and Journals for the Small Medical Library” [2], and Doody's Core Titles. In 2002, the MML undertook a project to identify a core medical journal collection for a post-Internet academic medical library. By analyzing print journal subscription lists of eighteen community-based medical school libraries and by entering these and selected quality indicators into a Microsoft Access database, the researchers created a core list. In addition to SERHOLD data from the community-based medical school libraries, other criteria included appearance on the “Brandon/Hill Selected List of Print Books and Journals for the Small Medical Library,” inclusion as one of the journals scanned for the American College of Physicians (ACP) Journal Club or the InfoPoems Clinical Awareness System, journals included in the British Medical Association's Core Collection of Medical Books and Journals, and the journals most frequently borrowed through DOCLINE nationwide for the calendar year 2000.

Based on a 10-point scale, a total of 1,442 core titles was identified and segmented into 3 levels or tiers [3]. Tier 1 journals received 3 or more points on the 10-point scale; 449 journals are in tier 1. Tier 2 journals received 2 points; 457 journals are in tier 2. Tier 3 journals received 1 point; 536 journals are in tier 3. The resulting core list, including all 3 levels or tiers, has been used from its creation to assess the adequacy of currently available campus-wide resources required to support the needs of a new college of medicine and to evaluate electronic journal packages for cost-effectiveness when making licensing decisions. The current study was undertaken to evaluate the usage of the titles on that list after 5 years of experience using it. Table 1 provides detailed information on the number of journals at each point value.

Table 1
Point distribution for the 1,442 titles in the core list

Two recent articles correlated use of electronic journals to appearance on core lists. An article by Wulff and Nixon correlated use of electronic journals at the University of Louisville to appearance on the Brandon/Hill list as well as to use of the printed versions [4]. A similar article by McKibbon et al. highly correlated use by practicing clinicians of electronic journals selected for inclusion in the ACP Journal Club and of journals selected for review by the producers of InfoPoems [5]. However, participants in the McKibbon study were provided access to only 348 MEDLINE journals; FSU COM users, on the other hand, are able to access more than 2,500 MEDLINE journals along with hundreds of other journals in related disciplines through FSU University Library subscriptions. The present work built on the works of Wulff and McKibbon through a focused approach to data collection that allowed for distinguishing e-journal use by a stand-alone college of medicine from use by other university populations and another opportunity to correlate a core journal list with use of electronic journals. As noted earlier, the FSU COM core list was created with a community-based medical school population in mind from the start. Therefore, gaining knowledge of how effectively the core list met the needs of the FSU COM went beyond previous studies that were limited to 2 or 3 general core lists such as the Brandon/Hill list.

This article reports on an evaluation of the FSU COM core list that compares actual use of a subset of the journals by both COM and university users with the weighted point value assigned to each journal in the core list. This evaluation includes a review of four e-journal packages and an assessment of their effectiveness in addressing the needs of the FSU community-based medical school for peer-reviewed journals. Several questions asked during this evaluation will be answered throughout this article and will appear as conclusions:

  1. How well does each journal package meet the distinct needs of the COM compared with the larger university at FSU, and how might an understanding of both the weighted values of the FSU COM core list and overall use patterns inform selection decisions?
  2. How well do the individual point values of the FSU COM core list correlate to actual use by COM users?
  3. Are there journals that are not on the core list that should be considered for addition based on usage criteria?
  4. Can areas of discipline overlap be identified between the FSU COM autonomous academic medical library and the larger university?


Four journal vendors were contacted and asked to collect and report full-text use of journal articles by Internet protocol (IP) address beginning in late 2006. For the purpose of this study, the researchers decided to evaluate and present data for one full calendar year beginning in January 2007. The COM has its own EZproxy authentication software, with a configuration that is discrete from the one that serves the rest of the university, and manages its own usernames and passwords; thus, use by COM users can be separated from use outside of the college.

The four packages making up the studied subset were: Blackwell-Synergy (Blackwell), Cell Press (Cell), Ovid's Lippincott Williams & Wilkins (LWW), and Elsevier's ScienceDirect (SD). Usage data were sent by vendors or extracted from vendor statistics web portals in February 2008, loaded into an Excel spreadsheet, and core list values (0–10 points), based on the weighted values assigned in the earlier study, added to each journal in the spreadsheet. All studied journals were available as part of site-wide licenses with unlimited access. Therefore, a “license-exceeds” scenario could not occur, and collected data included all accesses for these titles on these vendors' sites.

All LWW package journals, all Cell journals, and medical or biomedical Blackwell and SD journals were included in the overall analysis, making up a total of 896 journal titles. Four-hundred and eighty-six core journal titles, or one-third of the total of 1,442 FSU COM core list journals, were the focus of the study. The total number of reviewed journals and their distribution on the tiers of the core list is shown in Figure 1. The reader should recall that tier 1 journals received the highest number of points (3–10 points) and are therefore considered the most important to the FSU COM.

Figure 1
Core list tiers for journals by package

Numerous factors complicated the analysis. Journal movement from vendor to vendor, title changes, new journal titles, and alternate access paths to the four vendor packages made it necessary to place more emphasis on process than on absolute results. For example, several journals were available both from SD and MD Consult, but IP-address-specific data were not available for the MD Consult–accessed titles. Using the data from each of the four packages, a number of charts, graphs, and lists were developed, describing use of these journals by the COM and by the university, the cost of these journals if purchased a la carte or as part of a “big deal” package, and the average core list point values for journals receiving various levels of use. The latter was developed to determine if the core list rankings correlated to actual use, that is, if the most used journals also received the highest point values. The charts were studied for trends and themes. Selected charts are presented throughout this paper to illustrate one library's process in making collection decisions.

Tables 2 and and33 list the journals most used by the COM and the journals not appearing on the FSU COM core list but accessed 50 or more times by COM users, respectively. Journals appearing on the core list but seldom used require further study before consideration for removal from the list; lack of use on one publisher's site could just mean that the journal had moved from one vendor to another. Specific cost information is not provided in this article because, owing to the many peculiarities of local and statewide arrangements with publishers and vendors, costs vary from library to library. Naturally, cost considerations must be factored in when reviewing the data and when making local decisions on the cost-benefit ratio of each package. Each library must develop its own cost-benefit analysis based on its own institutional environment.

Table 2
Journals used more than 200 times by Florida State University College of Medicine (FSU COM)
Table 3
Titles used 50 or more times by the COM for consideration as additions to FSU COM core list


Journal use

Figure 2 compares percentage of use for titles in each of the three core list tiers by vendor or publisher, for both the university and the COM user communities, as well as percentage of total use by tier for both the university and the COM user communities. Tier 1 journals were considered of most value to the COM because these journals received 3 or more points out of 10 points in the original study; tier 2 journals were considered of moderate value to the COM because they received 2 points out of 10 possible points; and tier 3 journals were considered of lowest value to the COM because they received only 1 point out of 10 possible points. The total number of articles accessed in all of the 896 journals during the period of study was 162,169. The total number of articles accessed in any of the 486 journals in this subset that were on the core titles list was 122,533.

Figure 2
Comparison of COM and university use of journals in each package by core list tier

Figure 3 shows the distribution of the 486 core list journals according to levels of use by the COM. Median core values for each usage level were calculated, and median core point values are presented for 3 of the 4 packages by level of use. Because there were only 4 core Cell journals included in the study, it was determined that a separate analysis by level of use was not necessary or useful for this package. Each category of use (journals accessed 200 or more times, journals accessed 100–199 times, journals accessed 50–99 times, journals accessed 25–49 times, and journals receiving 0–24 uses) includes the average core point value for each package and for all packages in aggregate.

Figure 3
COM use of core journals grouped by usage with median core values for each vendor package (n = 486)

This figure indicates that usage of journals with higher core list values experienced greater usage. A statistical analysis was done using nonparametric statistical methods because the data were not normally distributed and ordinal values were used. A moderate correlation was found between point values and usage, represented by a Spearman's-rho value of 0.449. This correlation is significant at the 0.01 level (2-tailed).

The 189 journal titles in tier 1 received an average of 92 uses per title by the COM; the 145 journals in tier 2 received an average of 56 uses per title by the COM; and the 161 journal titles in tier 3 received an average of 26 uses per title by the COM. As noted earlier, these results must be reviewed with the understanding that real life events muddied the analysis: several journal titles migrated from one vendor to another during the study period, and some journals were also available from other vendors who did not provide use data by IP address.

College of medicine and university use

An overview of use by all 896 journals in the 4 packages, not just the 486 core journals studied above, provided an opportunity to compare COM and university use patterns. Figure 4 presents the number of titles in each level of use (heavily used to little used journals) for each user group. Figure 5 presents the total amount of use of these journals for each level of use again by the COM and by the university.

Figure 4
Distribution of journal use by COM and university users
Figure 5
Distribution of use by COM and university users

Though identifying areas of common interest across the university was not one of the primary objectives at the outset of this project, it was of some interest. By selecting journals that received roughly equal use by the COM and the general university, or around 50% each, the librarians were able to identify several areas that represent opportunities for multidisciplinary resource sharing and collaboration. If a journal was used heavily by the COM (defined as used over 50 times) and the COM use represented between 45% and 55% of the total use by both the university and the COM, the journal title was selected for review. Table 4 provides a list of journals that represent such areas of overlap. Topic areas included: cell biology, obstetrics and gynecology, pediatrics, and public health.

Table 4
Journals receiving similar levels of usage by FSU and COM

Cost effectiveness

In addition to recording the number of journals in each package that appeared on the COM core list, the librarians evaluated each vendor package for cost-effectiveness based on usage by the COM. Cost data are not presented here; however, as an example of how usage data factored into this approach, Table 5 provides data on the number of uses in 320 SD journals expressed as totals and cumulative percentages for the SD journals. A small percentage of journals (5%) represented 30% of use by the COM; 14% of the journals received 50% of use; and 41% of the journals received 80% of use. This information, along with cost information for journals if purchased on an a la carte basis or as part of a package deal, is useful in assessing the value of the SD package to the COM.

Table 5
ScienceDirect use by COM

Additions to the core list

The FSU COM license to LWW provided access to all journals in the package; individual titles could not be dropped to conserve funds. Use of these titles was high at 14,230 for the COM and 18,430 for the university, with a cost-per-use ratio in line with other journal packages studied by the COM MML librarians. Tier 1 LWW journals (journals receiving between 3 and 10 points) received heavy use by COM users at 5,466 uses; however, journals not in any tier of the core received 5,812 uses by COM users or 41% of the use overall. One unexpected finding was that roughly 20% of the use of nursing journals in the LWW package was made by the COM. As a result of this analysis, 24 titles in the LWW package were identified for consideration to add to the core list and are listed in Table 3 along with proposed additions from the other packages studied. The 332 SD evaluated journals received 91,690 uses for both the COM and the general university combined. Ten titles were identified for consideration to add to the FSU COM core list. A total of 258 Blackwell journals were included in the study. The COM accessed only 4,324 Blackwell articles compared with 25,078 used by the university. Tier 1 journals, that is, the most core of the titles, received the heaviest use by the COM, followed respectively by tier 2 and tier 3 journals. Two Blackwell titles studied were identified for addition to the core list. The Cell journal package was determined to be very cost efficient, with 6 journals receiving 8,417 uses overall across the university. Four of the journals received more than 100 uses by the COM. One journal was identified for addition to the core list.


One of the primary objectives of this study was to determine if a correlation existed between appearance by a journal on the FSU COM core list and actual use by the COM population. If this were the case, use of the FSU COM core list could be considered one valid way to evaluate a journal or journals for inclusion or exclusion when making collection decisions in similar medical library environments. The analysis revealed a moderate, but significant, correlation.

The data were also analyzed for the suitability and cost effectiveness of specific journal packages for the FSU COM. It is important to remember when reviewing these data that the COM library was created in a university environment, one that includes life sciences, human sciences, and social sciences research communities already supported by a number of electronic journals provided by the FSU University Library when the COM started up. The results might not, therefore, translate precisely to other environments. However, this evaluation and its conclusions should provide some methods to assess the value of a given e-journal package, even in other settings, and to help identify journals with a more clinical focus. For example, when a price tag is attached to each of the most highly used journals as well as to journals in each tier of the core list, it is possible to calculate the cost of the group of highly used journals if purchased a la carte as well as the cost of tier 1 and tier 2 journals if purchased a la carte. This information is helpful in evaluating both a la carte selections and package deals. In the case of Blackwell, if purchasing for a small or stand-alone medical school, a la carte might be the best course of action. In a large multidisciplinary university, a package deal might be preferred. However, the best decisions will require that dollar amounts are added to each of these two views (most heavily used journals and high-point-value core list journals) for individual institutions.

The data on use of journals by level of use demonstrated in Figures 4 and and55 provide clear evidence that the COM has more focused needs than the university. While this may seem obvious, knowledge of absolute levels of use by each journal is helpful in estimating the value of journals if priced on an a la carte basis or as part of a big deal. It is also helpful to know how the total use of journals by COM users compares with the total use of journals by university users. The authors are not aware of any other reports in the literature that provide such a large sample of use data collected in a diverse university environment that separates COM use from university use and uses these data to evaluate recommended or core medical journals.


In closing, the authors return to the four questions posed at the beginning of the article with the knowledge that each library must develop its own set of criteria on which to base selection and retention decisions. More than ever, libraries must function within a larger institutional, state, or regional purchasing environment. It is expected that this article will raise some provocative questions for each individual library as it goes in search of its best deal in the environment in which it exists. The authors hope that the FSU COM core list and this follow-up evaluation of use patterns by both a college of medicine and a larger university provides a strategy that will assist readers in exploring ways to use the core list and usage data in their own institutional setting.

1. How well does each journal package meet the distinct needs of the college of medicine (COM) compared with the larger university at Florida State University (FSU), and how might an understanding of both the value of the FSU COM core list and overall use patterns inform selection decisions?

Though it includes some journals that are not relevant to COM users, the LWW package is heavily used and provides a reasonable cost-per-use-ratio compared with other FSU COM subscriptions. LWW meets both COM and university needs and seems to be of equal value to both the COM and the university. Nursing journals received more use by the COM than the librarians expected.

SD medical journals receive heavy use by both the university and the COM, indicating that the university as a whole benefits greatly from this package in meeting both medical and biomedical research needs. One way to evaluate this package for value for the dollar is to compare the cost of a la carte selections for heavily used journals and for highly valued journals, based on appearance on the FSU COM core list, with other pricing options that the company may offer to individual institutions or consortia.

COM use of Blackwell journals is limited and is dispersed across all core list tiers. Forty-eight journal titles that were used 25 or more times account for 68% of the use by the COM of the Blackwell titles. Currently, many of the Blackwell titles are included as part of a bundled package. If the package is discontinued for any reason, the COM would consider a la carte licensing for these titles.

Cell journals receive heavy use and are used by both the COM and university equally.

2. How well do the individual point values of the FSU COM core list correlate to actual use by COM users?

There is a positive correlation between journal usage and core list values. Because the correlation is moderate, the FSU COM core list should be used in conjunction with local usage data and user input when making collection decisions.

3. Are there journals that are not on the core list that should be considered for addition based on usage criteria?

Thirty-seven journals were used fifty or more times and should be considered for addition to the FSU COM core list. These are identified in Table 3.

4. What are the areas of discipline overlap between the FSU COM autonomous academic medical library and the larger university?

There are several areas based on use data that suggest discipline overlap at FSU: cell biology, several areas in clinical medicine, public health, and nursing. Additional study may help identify other common research interests.


Many hours were required of our staff and colleagues—Nadine Dexter, AHIP, Carol Warren, Roxann Williams, and Wash Anderson—in supporting the authors throughout the lengthy data collection and analysis project. Of special note is Emiko Weeks, senior library assistant, who worked diligently and with good humor throughout on the data reporting and checking phase. We cannot thank her enough for her hard work and care.


An external file that holds a picture, illustration, etc.
Object name is mlab-irp.jpgThis article has been approved for the Medical Library Association's Independent Reading Program <http://www.mlanet.org/education/irp/>.

*Formerly, head of technical services, Maguire Medical Library, Florida State University.


1. Burrows S. A review of electronic journal acquisition, management, and use in health sciences libraries. J Med Libr Assoc. 2006 Jan;94(1):67–74. [PMC free article] [PubMed]
2. Hill D.R., Stickell H.N. Brandon/Hill selected list of print books and journals for the small medical library. Bull Med Libr Assoc. 2001 Apr;89(2):131–53. [PMC free article] [PubMed]
3. Shearer B.S., Nagy S.P. Developing an academic medical library core journal collection in the (almost) post-print era: the Florida State University College of Medicine Medical Library experience. J Med Libr Assoc. 2003 Jul;91(3):292–302. [PMC free article] [PubMed]
4. Wulff J.L., Nixon N.D. Quality markers and use of electronic journals in an academic health sciences library. J Med Libr Assoc. 2004 Jul;92(3):315–22. [PMC free article] [PubMed]
5. McKibbon K.A., Haynes R.B., McKinlay R.J., Lokker C. Which journals do primary care physicians and specialists access from an online service? J Med Libr Assoc. 2007 Jul;95(3):246–54. [PMC free article] [PubMed]

Articles from Journal of the Medical Library Association : JMLA are provided here courtesy of Medical Library Association
PubReader format: click here to try


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...