Methods: Eighteen potential users were asked to attempt to answer clinical questions using five commercial products (ACP's PIER, DISEASEDEX, FIRSTConsult, InfoRetriever, and UpToDate). Users evaluated each tool for ease-of-use and user satisfaction. The average number of questions answered and user satisfaction were measured for each product.

Results: Results show no significant differences in user perceptions of content quality. However, user interaction measures (such as screen layout) show a significant preference for the UpToDate product. In addition, users found answers to significantly more questions using UpToDate.

One barrier encountered by physicians attempting to answer questions with evidence is lack of time to perform all of the steps outlined above [4]. In an observational study, participating physicians spent less than two minutes seeking the answer to a clinical question [5]. In addition to the lack of time, many clinicians may also lack the skills needed to search, analyze, and synthesize the primary literature [6]. While it could be argued that clinicians could or ought to learn these skills, other professionals already have these skills. In a modern team-based approach to health care, it is natural for others to do the information synthesis and analysis [7]. In the McColl study cited above, 37% of respondents felt the most appropriate way to practice EBM was to use “guidelines or protocols developed by colleagues for use by others.” Only 5% felt that “identifying and appraising the primary literature” was the most appropriate. When others do the searching and synthesizing, clinicians no longer need to assemble the evidence as the need arises. Access to these predigested forms of information readily supports the use of evidence in clinical decision making.

Many electronic products have been specifically designed to provide this sort of synthesized information to the clinician. For the purpose of this discussion, they can be called bedside information tools. Because they have been designed to provide information at the point of care, they tend to provide information that will directly affect patient care. Although like other secondary sources, they can be a good source for background information. What distinguishes these products from sources such as electronic textbooks is this emphasis on point-of-care, patient-oriented information. These products are distinctly different from text-based, point-of-care sources such as printed guidelines: these electronic resources can be updated much more frequently than print resources and provide tools for easily searching and sharing information.

As with other forms of secondary literature, a number of such resources are available from which to choose. With limited financial resources, libraries and other institutions cannot afford to subscribe to or purchase all available bedside information tools. Thus, providing access to all the bedside information tools is not a practical solution. The question then is which bedside information tools should be selected and supported by an institution or library? To make this decision, a method for evaluating these products is needed.

Academic medical and hospital libraries have long been tasked with selecting which information products such as books and journals to purchase and make available to their patrons. This task is often guided by a collection development policy, which states what topics and formats are collected and to what extent. These decisions are driven by the population that the library serves, including people such as researchers, clinicians, and students as well as departments, disciplines, and programs supported by the library. In addition, librarians have traditionally used review tools to aid their decisions. For example, the ISI's Journal Citation Reports contain citation analysis data, including ISI impact factors, often used in decisions about which journals to subscribe to [8]. Another commonly used tool has been the Brandon/Hill selected lists. Compiled by individuals using criteria such as content, authority, scope, audience, and timeliness, the Brandon/Hill selected lists have long been recognized as containing the “best” core books and journals available for medicine, nursing, and allied health. Although the Brandon/Hill selected lists have recently been discontinued [9], their rapid replacement by Doody's Core Titles [10] is testament to how essential librarians have found these tools for collection development.

The evaluation of bedside information tools poses a challenge, as few tools are available to assist the evaluator. Product reviews are sparse. Published reviews were found for each of the five products evaluated in the current work; however, more than one review was found for only four products [11–21]. With frequent updates and changes to electronic information resources, published reviews may quickly become out of date. Often these reviews are only for the personal digital assistant (PDA) version of the product or similarly limited in scope. In addition, published articles about a product are often authored by those involved with the development of the product itself [11, 22–26]. There are no centralized reviewers, as with the Brandon/Hill selected lists, or generally accepted metrics, such as ISI impact factors. While the Healthcare Information and Management Systems Society Solution Toolkit [27] is a potential evaluation tool, bedside information tools generally fall outside of its scope; only two of the products evaluated here are included. Without readily available evaluations of these tools, librarians and other decision makers are forced to evaluate these tools themselves [28].

Evaluations of more traditional information products often focus on the product itself, as do many evaluations of information retrieval (IR) systems. The interactive nature of the bedside information products being evaluated requires that a user's interaction with the product be evaluated as well as the content of the product. User-centered evaluations look at the user and the system as a whole when evaluating an IR system [29]. Task-oriented evaluations ask users to attempt a task on the system, then ask if the user is able to complete the task. A task-oriented evaluation of bedside information tools asks the question, “Can users find answers to clinical questions using this tool?,” and seeks to answer the question by having users attempt to complete these tasks. By looking at both the system and the user, these evaluations measure whether or not an IR system does what it is designed to do.

User satisfaction assessment goes one step further and asks not only “Did the system do what it was designed to do?,” but also “Was the user satisfied with the way it did it?” By measuring user satisfaction, an evaluation can identify not only those products that successfully answer clinical questions, but also those products that the users enjoy using the most.

The finding that UpToDate is generally preferred by the users in this study is consistent with studies that tested what tools users select when they have access to more than one tool. These use studies, while not exploring the reasons that people use the products they use, do indicate that when given a choice, users pick UpToDate as a resource very frequently [48, 49].

Unlike some other studies [34, 38, 41], no training on the resources was provided. Users may have had different opinions about some products if they had been trained in their optimal use. However, users were allowed to familiarize themselves with the product before the evaluation if they wished. It can also be argued that sitting down to a resource with no training whatsoever is standard practice; this study attempted to replicate that practice. If a product requires training for users to be successful with it, those training costs must be considered when comparing to other options. An inexpensive product that requires extensive training may be more expensive in the long run than a more expensive product that requires little or no training.

Although thorough efforts were made to recruit a broad sample of potential users, the evaluators were essentially a convenience sample of volunteers. These volunteers might not represent the total user population as they tended to have an interest in these types of products at the study's outset. Their responses may not be typical of other users who, having no interest in the technology, are still “forced” to use these types of products for one reason or other. Another result of relying on pure volunteerism is that the users in various classes were not evenly represented. Input from other types of clinicians, including more nurses, physician's assistants, dentists, and others may have changed the results. Alternately, a more homogenous group could have given more reliable results. The current sample size also precluded comparisons by type of user or previous experience with these products.

From an initial volunteer sample of twenty-four participants, only eighteen completed the study in the time frame of this study. Participants who failed to complete the study did not differ from those who completed the evaluation in terms of recruitment method or clinician type (as ascertained from the initial meeting with these participants). Dropout rates might have been high because participants were asked to perform the evaluations on their own time. A laboratory-based evaluation might have reduced dropout rates but might have also reduced the number of volunteers in total.

In addition to the limitation caused by having a volunteer pool of participants, the products tested also amounted to a convenience sample of those products the OHSU Library was currently interested in. When compared to a different selection of products, UpToDate might not have ranked as high.

Although user evaluations in this study pointed to differences in user interaction scales, those that measured the user's perception of the content of the product (accuracy, currency, and amount of information) did not show any significant differences. Evaluations that focus on the content of an information resource, rather than on the user's experience, would not have revealed the strong user preference for UpToDate. These types of products must be evaluated in terms of user interaction as well as their content. For example, a product selected because of excellent content can be rendered useless by a difficult user interface.

As electronic formats become the expected form for information, as in the case of the products evaluated here, the interface takes on greater significance. In this study, users rated the content of these products to be equivalent but found significant differences in the user interface. In the case of these types of bedside information tools, the interface is the product as much, if not more, than the content of the product. This only underscores that content-focused evaluation methods may no longer be sufficient in evaluating this new breed of information tools.