The National Library of Medicine (NLM) plays an active role in the provision of health information to consumers. Medline-Plus , which was launched in October 1998, typifies the NLM’s commitment to providing trustworthy, well-organized health information [1]. The MedlinePlus critieria includes (1) the quality, authority and accuracy of content; (2) the primary purpose of the Web page (i.e. educational and not to sell a product or service), (3) the availability and maintenance of the Web page and (4) special features, for example providing content that is accessible to persons with disabilities¹.

The Go Local intuitive (www.nlm.nih.gov/medlinplus/golocal.html) augments information in MedlinePlus with local health services, programs, and health care providers. The first Go Local portal was funded in August 1999 at the University of North Carolina, Chapel Hill as a joint project between the Health Sciences Library and the School of Information and Library Science. The next portal was created at the University of Missouri, and since then Go Local portals have been initiated in Alabama, Arizona, California, Indiana, Maryland, Massachusetts, Michigan, Ohio, Texas, Tribal Four Corners (Arizona, Colorado, New Mexico and Utah), and Wyoming.

Several studies have explored the trustworthiness of health information on the WWW [2–4]. A cataloger on the NC Health Info portal fulfills two roles. The first is to ensure that the information resource (which we refer to as a web page) is trustworthy. Figure 1 captures the primary inclusion criterion used by catalogers for a Go Local portal. This inclusion criterion reflects both MedlinePlus and Go Local criteria.

In addition to trustworthiness, the catalogers on the NC Health Info portal assign terms from the Go Local controlled vocabulary. Although several projects have developed search engines specifically designed for health information, the controlled vocabularies in both MedlinePlus and the Go Local portals still play an important role in enabling a user to identify health information on the web.

Figure 2 provides an example of the cataloger’s annotations for web page 46. On May 31, 2002, a cataloger working on the NC Health Info project established that web page 46 satisfied the authority, content, audience, and local relevance criterion shown in Figure 1. On the same day cataloger AA posed a question about the scope of the geographical location of this service (line 2), and on June 11, 2002 cataloger BB responded to AA’s question (line 3). In September 2002, cataloger CC updated the geographical locations that were associated with the web page (line 4). The page was again updated in March 2003; however, the cataloger information for that update is not available (line 5). In April and October 2003 (line 6 and 7), the system proposed that the site should be reviewed, but the catalogers found that no changes to the database record were necessary. In April 2004, DD again reviewed the site with respect to the geographical location and posed a question (line 8), which cataloger FF answered (line 10) three days later.

The example database records shown in Figure 2 demonstrate the rich set of interactions that occurred between at least six catalogers over the two year period. Annotations 2–5, 8, and 10 capture interactions between catalogers as they work through the process of assigning the initial health services terms and topics, and as they continue to maintain the web pages. The maintenance schedule for the NCHealthInfo project is currently every six months. The system generated annotations 1, 6, 7 and 9 automatically.

In general, the kappa statistic is used to report inter-rater reliability [5]; however, studies of on-line web pages have shown that researchers rarely provide inter-rater reliability [3]. The example in Figure 2 demonstrates that catalogers seek consensus before assigning a controlled vocabulary term. Thus, the kappa statistic would not provide insight into the nature of the interactions shown in Figure 2 because disagreements are resolved as part of the cataloging process.

In contrast to the kappa statistic, a qualitative analysis can provide insight into the challenges faced by catalogers as they assign terms from the Go Local controlled vocabulary to each web page that satisfies the quality criterion. We used content analysis to characterize the nature of disagreements, such as the discussions shown in Figure 2.

Table 1 captures the content, format and function facets that we considered during the content analysis. These facets and categories were developed from our initial pilot study that comprised a random sample of 464 web pages (20%) from the web pages in the NC Health Info database. Removing non-substantive annotations yielded 371 substantive messages. Two of the authors (LL and DW) characterized each of those 371 annotations, and Table 1 shows the categories that emerged. Once inter-rater reliability was established between the two authors, they labeled the remaining pages using these eleven categories shown in Table 1. The categories in each facet are not mutually exclusive, thus any given substantive annotation can have multiple categories assigned. Every annotation has at least one category for each facet.

In order to understand if annotation behavior of catalogers changed over time, we compared the number of annotations for six different catalogers from April 2002 to September 2004 (data not shown). Our intuition behind this analysis was that the number of annotations made by an individual cataloger would decrease as their familiarity with the cataloging process increased. Such an analysis might indicate the time required to train a new cataloger. Contrary to our expectations experienced catalogers continued to provide annotations throughout the cataloging process.

Annotations enabled the catalogers to form consensus around the meaning of an existing information source, an activity that is not new in medicine. Scientists who conduct systematic reviews develop extraction worksheets that capture their consensus building activities [6]. Multiple reviewers independently extract information using the guidelines, then resolve differences. This process enables the group to establish group norms and verify the accurate extraction of information from each article.

Similarly, consensus building is an important consideration in recent efforts in bioinformatics to annotate scientific articles with terms from the gene ontology (www.geneontology.com). In both the systematic review and bioinformatics examples scientists have developed hierarchies of evidence that reflect the annotator’s confidence in the final category assignment. In a systematic review, the stated study design reflects the level of evidence while in bioinformatics, scientists have invented a set of evidence codes³ including “inferred from assay” and “inferred from genetic interaction” to measure their confidence. The annotations provided by the NC Health Info catalogers also reflect the cataloger’s confidence in the final annotation. This serves as a surrogate for levels of evidence in this new area of web page annotation until accepted levels of evidence are developed.

The large number of annotations related to topical scope suggests that additional conversations are required to define the boundary of an online information resource. In this paper, we have hidden the complexity regarding annotating online information resources, by referring to each resource as a web page, which implies an individual page. However, catalogers do not catalog every individual web page on a site; rather they catalog an entire site, or a sub-set of pages within a site. The number of annotations indicates that defining these boundaries is very challenging.

Two of the challenges faced by catalogers are specific to an on-line environment. The first concerns the web page boundary, for example when should the cataloger assign a topic to an entire web site, and when should they assign topics to subdomains? The second issue concerns the dynamic nature of on-line information compared to traditional information resources. Currently catalogers review web pages in the NC Health Info portal every six months. However, a cataloger need not review an unchanged web page, and should conduct a review if the page has changed within the six-month period. Thus, an information system that detected change within a web page would aid in the allocation of resources for the review task.

None of the individuals who worked on the NC Health Info project was required to annotate their cataloging process; yet they provided annotations for more than half of the web pages. This suggests catalogers find annotations useful. Many of the catalogers on this project were students, so these annotations have long-term implications with respect to preserving organizational memory.