Methods: From a pool of 33,449 citations in 934 articles published in the 2003–2005 issues of American Journal of Public Health, 2 random samples were drawn: one (n = 1,034) from the total set of citations and one (n = 1,016) from the citations to journal articles. For each sampled citation, investigators noted publication type, publication date, uniform resource locator (URL) citation (yes/no), and, for the journal article sample, journal titles. The cited journal titles were analyzed using Bradford zones.

Results: The majority of cited items from the overall sample of 1,034 items were journal articles (64.4%, n = 666), followed by government documents (n = 130), books (n = 122), and miscellaneous sources (n = 116). Publication date ranged from 1826–2005 (mean = 1995, mode = 2002). Most cited items were between 0 and 5 years old (50.3%, n = 512). In the sample of 1,016 journal article citations, a total of 387 journal titles were cited.

One strategy librarians have employed in other multidisciplinary fields is citation analysis [3,4]. Citation analyses and bibliometric studies of public health have largely focused on two factors: impact factors and authorship trends [5–8].

Public health's subdisciplines have undergone more extensive bibliometric analysis than the general field of public health. Several recent studies have measured various countries' output in virology [9], microbiology [10], parasitology [11], and infectious diseases [12,13]. Other bibliometric investigations have considered the public health subdisciplines of prevention research [14], health education [15], occupational health [16], tropical medicine [17,18], epidemiology [19], public health nursing [20], and health care management [21].

At the start of this research, the researchers downloaded ISI Web of Science data for all journal issues of the AJPH published in 2003–2005. This initial download resulted in 1,299 citing items, of which 908 were designated “articles” or “review” articles by ISI, as the authors decided to omit letters, editorials, and historical pieces. These 908 articles yielded 30,695 cited items according to the ISI data. On closer inspection, however, the researchers discovered that a large number of cited items were missing from the data, in particular “unusual” publication types such as Web resources and court cases. Due to these omissions, the researchers instead manually built a data set of all qualifying articles by hand-counting and sequentially numbering the citations in each article.

The manual review of all 37 issues of the AJPH from 2003–2005 resulted in 934 citing articles that yielded 33,499 cited items. Letters, editorials, editor's choice, other departments, “Faces of Public Health,” “Images of Health,” and “Voices from the Past” articles from AJPH were not included in the pool of citing articles. These categories were excluded for a variety of reasons. For example, the researchers felt that letters to the editor and editorials, because they often self-cite, could skew the results. “Voices from the Past” are article reprints, generally from the 1960s or earlier, and thus did not reflect current public health information needs.

Of the overall pool of cited items, 21,397 (63.9%) were journal articles. Because of the dual purpose of this study—to determine the publication types cited most often in public health as well as the most influential journal titles—the researchers next drew 2 samples. The first, a sample drawn from the overall pool of 33,499 cited items, was intended to yield data regarding the most frequently cited publication types and their age at time of citation. The second, a sample drawn from the 21,397 cited journal articles, was intended to provide data describing the most often cited journal titles and the breadth of journals consulted in public health research. Upon consultation with a statistician and using a sample size generator, it was determined that to achieve a 95% confidence level with a ±3% confidence interval, 1,034 items would be sampled from the complete cited item set; a second sample of 1,016 items was drawn from the set of cited journal articles [25]. Items were drawn from each sample randomly using an online random number generator [26]. Because each sample was drawn randomly without reference to the other, journal article citations had a random chance of being selected for both samples.

Collected data included publication type, publication date, and uniform resource locator (URL) citation (yes/no) for the cited item data set. Publication types were categorized as journal articles, books, government documents, or miscellaneous, the four categories Schloman used in her study of the health education literature [15], a public health specialty. Government serials, except for statistical publications such as the National Center for Health Statistics' Vital and Health Statistics Series, were coded as journals. Miscellaneous data included public laws, court cases, conference proceedings, and dissertations, among other things. Websites without a print-equivalent book, journal, or government publication were also coded as miscellaneous.

Cited item age was calculated by subtracting the cited item publication date from the citing article publication date. Data from the cited journal article sample included only the full journal title and the publication date. In all cases, the most current title of a journal was used for analysis; older title data were updated to current titles. All data were analyzed with SPSS software using cross-tabulations and the chi-square statistic with a P-value threshold of 0.05.

The cited items ranged widely in age, though most citations were from the most recent 5 years. Comparing this result to other citation analyses was difficult due to the wide variance in how citation analyses report citation age, but Rethlefsen's local citation analysis of Minnesota Department of Health (MDH) publications showed an even larger percentage of citations to materials from the most recent 5 years [30]. Though recent materials were favored, there was still some reliance on the past; over 7% of citations were 20 years old or more. For books and government documents, this trend was stronger: over 10% of citations to these formats were more than 20 years old.

Interestingly, relatively few citations (n = 69, 6.7%) included URLs. It should be noted that no journal article citations listed an URL, most likely due to standard reference format conventions regardless of delivery method. Certainly more than 69 cited items have an online equivalent when journal articles are included, yet this figure is strikingly dissimilar to the results from Rethlefsen's 2003–2004 local citation analysis, in which 13% of citations were for Websites [30]. Alpi and Adams's public health nursing study revealed an even smaller amount of Web citations, 1.4%, though their study examined citations from an earlier time period (1998–2000) when Web citations might not have been as prevalent [20]. Taylor et al.'s 2002–2004 health care management citation analysis revealed a similarly small percentage of Web materials (3.4%), however [21]. This finding is noteworthy because librarians and public health agencies continue to develop new Web resources for public health practitioners. Practitioners may be relying more on the published journal literature and print materials than freely available Web sources when preparing publications. Physical resources may thus continue to be critical to this community.

Cited journals also displayed patterns similar to other published citation analyses. A few titles garnered a large amount of citations, while a large number of titles were cited. Zone 1 journals included journals from public health (n = 8) and general medicine (n = 6). Other public health citation analyses have also shown that general medicine titles are important to public health research. For example, JAMA and New England Journal of Medicine have also appeared in the Zone 1 lists for public health nursing, health education, health care management, and epidemiology and the MDH citation analysis [15,19,20,21,30]. Of the Zone 1 titles, all were identified by the Core Public Health Journals project, version 2.0, as core titles for public health, although only eleven were designated Essential Core titles [31]. Though there were similarities between this research and other public health citation analyses, unique differences were visible in Zone 1 titles. Table 6 lists the Zone 1 titles in the current study, as well as Alpi and Adams's public health nursing [20], Schloman's health education [15], Rethlefsen's health department [30], Taylor et al.'s health care management [21], and Hasbrouck et al.'s epidemiology [19] citation analyses. All studies show marked variation in Zone 1 titles, particularly by degree of specialty publications appearing in Zone 1.

Over half of cited journal articles were from journals in fields other than public health. This finding echoed earlier studies of public health subdisciplines [16,17,19]. Others have noted that searching multiple databases is necessary to cover critical public health specialty journals [15,20,32]. Indeed, even the Zone 1 titles in this study show that searching multiple databases and even hand-searching is necessary; for instance, MMWR is not fully indexed by any common biomedical literature database, including PubMed, Science Citation Index, or CINAHL [20]. Based on coverage findings from the Mapping the Literature of Nursing Project, for the Zone 1 titles alone, PubMed and Science Citation Index provide the best, albeit incomplete, coverage [22,33]. The multidisciplinary nature of public health research, its dependence on literature from disciplines outside public health, and its heavy use of government documents and gray literature all make comprehensive searching in public health complicated and resource intensive.

This study's results are based on a randomly drawn sample from an initial sample of public health literature, AJPH articles from 2003 to 2005. Citation analysis based on one to five core journals from a discipline or specialty is a common practice, but drawing a sample is uncommon in bibliometric studies, which often rely on hand-counting and manual entry of vast amounts of data into spreadsheets or databases. Sampling has been done in studies of citations culled from theses and databases [34–37], however.

In this study, even though sampling made dividing the journal article citation results into zones difficult, the samples did correlate with the larger set. The observed ratio between cited journal articles and total citations matched the expected ratio using a chi square goodness-of-fit test with a significance level of 0.001. The samples drawn for this study, though planned to provide a statistically accurate result, did place restrictions on the analysis of cited journal titles. Most importantly, because of the small set, it was not possible to delineate accurately between Zone 2 and 3 titles in a Bradford distribution.

One of the more interesting observations of this study was the finding that ISI Web of Science data were incomplete for many types of cited material, including legal citations and Websites. Without these citations, the results would have been skewed toward journal articles and books and away from gray literature sources. Using automated data sources for bibliometric analysis, particularly in gray literature–dependent fields, should thus be viewed with great skepticism.

The Core Public Health Journals Project, version 2.0, lists 31 journals just in the Essential Core portion of its Key Journals for All Public Health list [31]. The JCR lists 99 titles in the Science edition's Public, Environmental, and Occupational Health category [28], 28 of which an earlier citation analysis by Soteriades and Falagas counted as “public health” [8]. Using so many source titles produces an overwhelming amount of citations to examine, particularly because the traditional online source of bibliometric information, ISI Web of Science, may not always give accurate citation counts, much less accurate citations. Examining the results of this study and the other public health citation analyses displayed in Table 6 shows that only 2 titles were included in Zone 1 in every study: JAMA and New England Journal of Medicine, neither of which is a public health journal.

This study contributes to the body of information known about public health information use and is intended to be a starting point for citation analysis of this subject area. As indicated by the comparison of the results of this analysis with other citation analyses of public health subdisciplines and the local citation analysis performed at MDH, core journals in public health will likely vary based on local needs and specialties. A great deal of further research is needed, not only to analyze the literature of public health disciplines, but also to examine differences in academic- versus field-oriented publications. The authors hope that this preliminary research will inspire others to continue bibliometric research in public health. Sampling large data sets makes such citation analysis more manageable, though using Bradford zones is not as clear-cut. Larger analyses are more likely to have more clearly delineated zones. In this smaller analysis, the break between Zone 2 and Zone 3 would have been arbitrary as the second third of citations ended in the middle of titles receiving two citations. However, even larger analyses struggle with establishing accurate zones and often must arbitrarily pick a zone for “border” titles and create less than equal percentages of citations between zones.