The value of library and information services in patient care: results of a multisite study *

Objective: The research conducted a large-scale, multisite study on the value and impact of library and information services on patient care. Methods: The study used: (1) 2 initial focus groups of librarians; (2) a web-based survey of physicians, residents, and nurses at 56 library sites serving 118 hospitals; and (3) 24 follow-up telephone interviews. Survey respondents were asked to base their responses on a recent incident in which they had sought information for patient care. Results: Of the 16,122 survey respondents, 3/4 said that they had definitely or probably handled aspects of the patient care situation differently as a result of the information. Among the reported changes were advice given to the patient (48%), diagnosis (25%), and choice of drugs (33%), other treatment (31%), and tests (23%). Almost all of the respondents (95%) said the information resulted in a better informed clinical decision. Respondents reported that the information allowed them to avoid the following adverse events: patient misunderstanding of the disease (23%), additional tests (19%), misdiagnosis (13%), adverse drug reactions (13%), medication errors (12%), and patient mortality (6%). Conclusions: Library and information resources were perceived as valuable, and the information obtained was seen as having an impact on patient care.


INTRODUCTION
The National Network of Libraries of Medicine, Middle Atlantic Region (NN/LM MAR), formed a planning group in 2007 to explore the possibility of replicating a landmark study on the value and impact of hospital libraries on clinical care, popularly referred to as ''the Rochester study'' [1]. The Rochester study was among the first to relate information services provided by librarians to patient care outcomes, and it has continued to be cited as evidence of the value and impact of library services.   (UNC). While the entire group planned the current study, the research itself was carried out by Marshall and her staff at UNC. The group used an approach known as ''Community-Based Participative Research'' (CBPR), in which practitioners and researchers are equal partners in the research process [2].
In 2009, Dunn et al. described the extensive planning process used for the current study as well as the results of the focus groups of librarians that were conducted during the planning phase [3]. Readers are referred to that paper for details of the Rochester study methods and measures and the updates to the current study made by the planning group, as well as references to earlier literature on the value and impact of library and information services in patient care. Since 2009, several more studies have been published, including the Sievert et al. study on the value of library resources to health care providers in two mid-continental states [4], the Jemison et al. study on return on investment in Department of Veterans Affairs (VA) libraries [5], the Aitken et al. study on clinical librarians at the point of care [6], and the McGowan et al. study on the impact of librarianprovided information in primary care [7]. The current study also drew on previous impact research conducted by Urquhart et al. in the National Health Service in the United Kingdom [8].

METHODS
The full study used multiple methods: focus groups of librarians for planning purposes; a web-based survey of physicians, residents, and nurses; and twenty-four follow-up, semi-structured interviews. The telephone interviews were used to gather more details from the participants on their information seeking and use experiences and their perceptions of the library and librarian. Readers should consult the Dunn et al. paper for results of the focus groups [3]. This paper provides an overview of the results of the survey, because it was the primary data collection instrument. Some findings from the interviews are included since they provide additional insight into the survey findings and the changing role of the library and librarian. The research questions in the current study were (1) whether the information resources used were perceived as valuable by physicians, residents, and nurses and (2) whether the obtained information was considered to have had an impact on patient care.

Survey
The survey began by asking the respondent's profession (physician, resident, nurse, or other), followed by what types of work their jobs involved (patient care, management/administration, clinical research, education, or other). Respondents could select as many types of work as applied; however, to proceed with the survey, either patient care or clinical research had to be among the types selected. This requirement increased the likelihood that the respondents would be able to answer the questions in the survey. Respondents were then asked to think of ''an occasion in the last six months when you looked for information for patient care that was not available in the patient record, electronic medical record (EMR) system, or lab results'' and to answer the survey questions based on that single incident. This approach, used in critical incident technique research [9], was intended to increase both respondent recall and the likelihood that responses would be directly related to patient care. The information search could have been conducted by the health professionals on their own or with the assistance of a librarian or library staff member.
Next, the respondent was asked to select a principal diagnosis of the patient to whom their situation related. The list of diagnoses was created by the study planning group based on a review of disease categories used by resources such as MedlinePlus, WebMD, and UpTo-Date. A question from the Rochester study was used to gather data on the type of needed information, for example, therapy, diagnosis, information for the patient, adverse effects, and so on. The respondents were then asked to select the specific resources that they used to answer their question. The Value Study Planning Group created the list of information resources based on their familiarity with library collections. Not all sites were expected to have all the resources available to their users. The survey proceeded to ask additional questions about each information resource used, including methods used to access the resource and where the search was conducted. Subsequent questions related to whether the respondents found the information they needed and whether the information saved them time. Outcome questions from the Rochester study [1] were used to measure the perceived value and impact on patient care. The survey is available in Appendix A (pages 24-32, online only).
The interviews provided an opportunity to gather more detail on the use of information resources as well as the roles of the library and librarian. Respondents were asked about their experiences with searching on their own as well as situations in which they asked a librarian for assistance. They were also asked whether there were other ways in which the librarian contributed to patient care or to the institution besides searching. The interviewer guide and questions are included in Appendix B (online only). A two-pass method was used in which codes assigned by the first coder were checked against those independently assigned by a second coder. The two coders discussed any differences and assigned a code based on consensus. Analysis was conducted using NVivo software.

Recruitment and sample
Initially, library sites were recruited within NN/LM MAR, which includes the states of Delaware, New The value of library and information services Jersey, New York, and Pennsylvania. Subsequently, the study was opened up to other sites across the United States and Canada. Each registered site was provided with a detailed facilitator handbook that described the steps in the study and the responsibilities of the facilitator and the researchers at UNC. The facilitator handbook is available as Appendix A (online only). The facilitator was asked to obtain one or more study champions who would support the study in their institution, provide a statement supporting the study that could be quoted in the invitation email, and cosign the invitation email with the librarian. In the email invitation, potential respondents were invited to ''participate in a study on the value of information in clinical settings.'' They were told that ''This survey is intended for physicians, residents and nurses who are involved in patient care or clinical research. It has the support of your institution's leaders and is being facilitated by your librarian. [Name of administrator] supports this study and encourages your participation,'' with a sample quote from an administrator: ''Please take time to participate in this study. Your answers to the survey questions may ultimately help our institution provide better patient care by supporting you with the information you need, when and where you need it'' (Appendix A, pages 19-20, online only). Ethics approval for the multisite study was obtained from the UNC Institutional Review Board (IRB). Copies of the UNC IRB application and approval were included in the facilitator handbook (Appendix A, pages 13-16, online only). Facilitators were offered support from the UNC research team if they were required to obtain a separate ethics approval from their own institution.
Considerable efforts were made by NN/LM MAR staff to recruit a diverse group of sites that included both urban and rural locations and different size settings. Library sites did not have to be members of NN/LM MAR to participate in the study. NN/LM MAR held a webinar to promote the study, and the opportunity was publicized at Medical Library Association chapter meetings in fall 2010. The site registration and support activities were handled by the UNC research team. Mentorship was also offered to facilitators at interested sites by members of the study planning group. Of the ninety-nine library sites that initially expressed interest, fifty-six sites eventually met all the requirements for study participation. The most common reasons for lack of participation were lack of administrative support from the institution, lack of time on the part of the library staff, and difficulty in obtaining ethics approval.
A pilot survey involving seven NN/LM MAR library sites was conducted in fall 2010. The full launch of the survey took place during spring 2011. The UNC researchers drew a stratified random sample of the pilot study respondents who had indicated that they would be willing to be contacted for a follow-up interview. If an interviewee declined to be interviewed when they were contacted or could not be reached, the next potential interviewee on the list was contacted. Respondents did not have to identify themselves in the survey unless they were willing to be contacted for an interview, in which case they were asked for their contact information. Each site was assigned a separate uniform resource locator (URL), which was included in the email discussion list or portal invitation. Respondents were asked to click on the link to access the survey.

RESULTS
The 56 participating library sites served physicians, residents, and nurses in 118 hospitals. All NN/LM regions except New England were represented in the study, as well as 4 sites from Canada. Twenty-three of the participating sites were located in states served by NN/LM MAR. Based on estimates provided by the study facilitators, the total number of physicians, residents, and nurses served by the 56 library sites was approximately 172,463. On that basis, the 16,122 responses represent a response rate of 10%. The response rate for physicians was 10% (n55,379), for residents 12% (2,123), and for nurses 7% (n56,788). Response rates by library site varied considerably: 21 sites had a survey response rate of 10% or less, 25 sites had a response rate of 11%-20%, and 10 sites had a response rate of 21% or more. Since only those who were (a) involved in patient care or clinical research and (b) could remember an event in the last 6 months where they had used an information resource could complete the survey, the actual response rate from eligible participants may be higher.
The demographic results showed that 63% (n55,992) of the respondents were female, which might be partially due to the relatively high number of nurses in the study. Most respondents were almost equally divided in 2 broad age categories, with 47% (n56,032) aged 25-44 and 47% (n56,032) aged 45-64. The number of years respondents had worked as health care providers varied, with 36% having worked less than 10 years (n54,623), 23% (n52,954) from 11-20 years, and 41% (n55,266) for more than 20 years. The higher proportion of respondents who had been health care providers for over 20 years may be due to the relatively small number of residents in the study (n52,123), compared to physicians (n55,379) and nurses (n56,788). Although efforts had been made to recruit smaller nonteaching hospitals and hospitals in rural areas, the majority of the sites were members of the Council of Teaching Hospitals (n546). A slightly higher number were located in an urban area (n548). The number of sites with over 500 beds (n530) was slightly higher than the number of sites with under 500 beds (n526).
To understand the nature of the needed information, respondents were provided with a list and asked, ''Which of the following best describes the principal diagnosis of the patient to whom your situation is related? (Select only one answer).'' The results shown in Table 1 (online only) show a wide range of principal diagnoses, with the most common being cancer and heart disease. Respondents were also asked, ''What type of information did you need to answer your question? (Select all that apply).'' As shown in Table 2 (online only), over half of the respondents checked therapy (54%), diagnosis (53%), and drug information (52%). Other types of needed information included clinical guidelines (48%), information for the patient (34%), adverse effects (29%), clinical procedure (29%), and patient safety (23). Only 6% of the respondents checked ''Other'' type of information.

Value and impact
The results for the variables used to determine the overall value of the information are shown in Table 3. The variables are organized around four themes: quality of the information, cognitive value, contribution to quality patient care, and time saved. The overwhelming majority of the respondents rated the information as relevant, accurate, and current, which suggests that the perceived quality of the information was high. The cognitive value of the information was also highly rated, as measured by whether the information refreshed the respondent's memory of details or facts, substantiated their prior knowledge or beliefs, or provided new knowledge. Similar results were obtained for the contribution to patient care indicators, which included what the potential useful-ness of the information in the future was, what its clinical value was, whether the information resulted in a better informed clinical decision, and whether it contributed to higher quality care. Overall, 85% of the respondents reported that having the information saved them time. The mean amount of time saved was 2.5 hours (median 1.0, SD7.8).
In a separate question, 3/4 of the health professionals (n510,303) said they definitely or probably handled some aspect of the patient care situation differently as a result of the information. The specific changes in patient care reported by the respondents are shown in Table 4.
In the health care system, avoidance of adverse events is particularly important. The adverse events avoided as a result of the information are shown in Table 5. A number of key outcomes related to patient safety such as misdiagnosis (13%), adverse drug reaction or interaction (13%), medication error (12%), and hospital acquired infection (3%) were all listed by respondents as outcomes that were avoided as a result of the information.
Respondents were asked to indicate the importance of the various sources they had used in their search. The 4 sources included information resources, diagnostic imaging, lab tests, and discussion with colleagues. Table 6 shows that the information  The value of library and information services resources were rated slightly higher than the others. The analysis was based on the number of respondents who reported using each source. In this question, 5% of respondents did not report using information resources; 31% did not report using diagnostic imaging; 22% did not report using lab tests; and 12% did not report using discussions with colleagues. Of the 5% (n5591) who did not report using information resources, 590 had reported using 1 or more specific information resources in an earlier survey question, which suggests that some respondents might have misunderstood this question.

Information resources used
As noted above, the list of information resources used in the survey was generated by the Value Study Planning Group, based on their experience with health sciences library collections. Not all sites were expected to have all the listed resources. Some caution should be exercised in interpreting the information resources data, since the health professionals were asked to base their survey responses on a single patient care incident. Nevertheless, the results do provide a picture of the type of resources used by a large number of health professionals when they searched for information related to a specific patient care situation ( Table 7). The 4 most frequently used resources were online journals (46%), PubMed/MEDLINE (42%), UpToDate (40%), and online books (30%). The overall rating for UpToDate was high due to the large proportion of residents who reported using this resource (77%). Print books and journals were also ranked in the top 10 resources used. Although CINAHL was used in 10% of the incidents overall, it was used substantially more by the nurses (18%). The data showed that nurses were using a wide range of the resources in addition to the nursing titles. While only 4% of the respondents reported using consumer health resources to answer their patient care questions, responses to another question about the impact of the information from all resources used showed that 48% changed the advice that they gave to their patient (Table 4). This suggests that the information found in multiple resources might be contributing to enhanced communication between health professionals and patients. When selecting the information resources used, respondents had the option of selecting, ''Not sure.'' Only 3% (n5477) of the respondents chose this option. When asked if they had found the information they needed, 59% of the respondents answered ''completely''; however, nurses (52%) were less likely than physicians (64%) and residents (63%) to answer ''completely.'' A greater proportion of nurses reported that they had only partially found the information due to time constraints. Responses to the question  about the physical location where the search was conducted showed that the respondent's office (51%) and the patient care unit (50%) were the primary locations, followed by the home (33%) and the library (13%) ( Table 8). Because librarians have put a great deal of effort into making their information resources and services available remotely, this result is not surprising.
In a related question on the access points used for the search, the institution's intranet (52%) and the library website (50%) were the most frequently cited (Table 9). More than a third (37%) reported using a search engine such as Google. Nineteen percent went to the library, and 14% asked a librarian or library staff member for assistance. While it is not guaranteed that the respondents were using the library-provided versions of all the information resources, there is a strong likelihood that they were doing so. The survey invitation made it clear that the study was on the value of library and information services in patient care and that it was being facilitated by the institution's librarian. Furthermore, respondents were told that their participation in the study would help the institution to ''provide you with the information you need, when and where you need it.''

Interview results
Although the focus of this article has been on the survey, the following interview highlights complement the survey results and provide some insight into the perceived role of the library and the librarian. Most of the 24 interviewees were physicians (46%), followed by nurses (29%) and residents (21%). The mean age was 44 years old ranging from 24-81 years old. Over half of the interviewees were female (54%). While the mean number of years worked as a physician was 23, with a range from 7-48, the mean number of years worked as a nurse was 10, ranging from 2-30. For residents, the mean year of their residency was 3, with a range from 2-5.
The themes that emerged in the survey results were echoed in the interview results. The majority of those interviewed indicated that the information they obtained was relevant, accurate, and current; would be of use in the future; was of clinical value; refreshed their memories of details or facts; resulted in a better informed clinical decision; contributed to higher quality of care; substantiated prior knowledge or belief; provided new knowledge; and saved them time. Interviewees reported that ready access to quality information resources helped them make  The value of library and information services better, safer, evidence-based decisions. The two quotes below indicate the impact on patient safety: I think mostly it is a safety thing, so [the information] makes me deliver safer care.
[The information] helps to prescribe the right medication or-or more importantly prevents prescribing the wrong one.
The following quotes relate the benefits of electronic access and the impact on quality care: I think that the easier the electronic access is to the current literature that the more likely it is that any treating physician is going to use that to alter treatment decisions and so patients benefit from that…I think patients benefit because their physicians are more up to date because being more up to date is easier to do now.
[Patients] end up having more educated people taking care of them. You know…people who are better looking at side effects, [who] better know how to manage them, better your understanding. I think that directly impacts patient care the more, the more informed [providers] are.
Interviewees commented specifically on the value of professional librarians in several areas. An example of librarians' contributions to the patient care setting is found in the following quote: And [the librarians] do a bunch of searches for us almost on a daily basis for our medicine morning report and they come to our weekly pediatric rounds so, I have lots of experience with them turning over searches for the residents and for me you know in…almost in real time. So it's, it's really a nice relationship.
In addition to direct patient care, some interviewees also commented on the value of the librarian's searching skills in research: I have used the librarian for advanced searches for research projects…it certainly has facilitated my ability to have more thorough searches for research projects and designing research projects which may alter the project. Important roles for librarians in the instruction of clinical staff were also evident, as shown in these quotes: The librarians teach my residents because I'm the program director, they teach my residents how to use all the resources that the library offers; all the different data tools including Micromedex and MD Consult, etc. So, so they're not just helping me directly, they're helping me indirectly because my learners are becoming more sophisticated.
The librarians have been very good at disseminating current information in terms of ways in which to access medical information to our residents. They do a number of teaching and courses, almost classes, for the residents in terms of ways in which to access medical information in the most appropriate manner and so I think that has direct implications for patient care in that it allows the residents to be much more efficient in, in gathering information.
Finally, there were examples of the librarian's value in supporting institutional performance improvement: I also serve on a couple of committees where we do quality assurance and performance improvement for specific disease categories. So sometimes in that setting getting help from the librarians to do a broader search for more…for a larger, more in depth topic rather than a particular patient is helpful because we can get that background data and use that to guide our performance improvement…when I go to a librarian it's not to help a particular patient, it's to help a patient category for a larger number of patients.
These results suggest that while the librarians are still valued for their intermediary search expertise, their contributions are becoming more diverse as they take on additional roles in education, research, and institutional performance improvement. The continuing use of print resources as well as the library and librarian are reminders that the profession is continuing to support information access in multiple formats and multiple access locations. What appeared least visible to the survey respondents and interviewees was the key role that librarians play in electronic information resources management in their institutions. When asked, ''Can you think of other ways [besides reference service, search assistance, and instruction, which had been probed in previous earlier questions] that your librarian(s) contributes to patient care in your institution?,'' none of the respondents specifically mentioned electronic information resources management. Neither did they mention the roles that librarians play in making information resources accessible from multiple locations and enhancing retrieval from search engines such as Google. This finding illustrates the challenges involved in communicating the role that librarians play in managing electronic information resources to users and administrators.

DISCUSSION
While the value and impact results in the Rochester study [1] are similar to those in the current study, there were many differences between the two. The current study took into account the changes in library services, health care, information technology, and research methods that have occurred in the intervening years. Thus, the current study focused on the use of information resources rather than asking the health professionals to request an online search from their librarian, as was done in the Rochester study.
Although the current study does not examine the role of the librarian as directly as did the Rochester study, the fact remains that librarians provide access to a wide range of electronic and print resources in their institutions and that this access was found to be highly valued. Respondents also reported that using the information resources with and without the assistance of a librarian had important impacts in specific patient care situations.
In the Rochester study, all 15 hospitals in a 5-county area participated in the study, lessening the possibility of site selection bias. Personal contact information was obtained for a systematic sample of physicians. Potential respondents were telephoned in advance to ask if they were willing to participate. Those who agreed were sent an invitation in the mail, followed by the paper survey. Full follow-up procedures by regular mail were used. As a result, the Rochester study response rate was 46.4% (n5208). In the current study, financial constraints plus the unwillingness of institutions to provide direct contact information for their physicians, residents, and nurses made an online survey the only option. On the positive side, using web-based survey software allowed the UNC researchers to customize the online survey for each participating site and to collect additional data on the information resources used. The researchers were also able to accept more sites into the study and include nurses as well as physicians and residents. This was an important addition, because nurses are increasingly important members of the health care team.
As mentioned earlier, the health professionals were asked to respond to the current survey on the basis of a single incident in which they used information resources to search for information related to patient care. The study is based on self-report by the physicians, residents, and nurses. Although the number of sites (56) and respondents (16,122) are both large, the results are not generalizable to all users of the information resources or to all patient care situations. It is possible that the health professionals who were motivated to participate in the study were more likely to be those who were interested in the library and its information resources. It is also possible that the respondents chose a clinical situation in which they had been at least partially successful in obtaining the information they needed. Alternatively, health professionals who did not have a successful search experience might also have been motivated to respond.
Currently, additional specialized reports based on the study data are being prepared by the UNC researchers. These reports will be made available on the NN/LM MAR website ,http://nnlm.gov/mar/ about/value.html.. The study planning group and NN/LM MAR are committed to disseminating the study results and materials as widely as possible so that other groups that are interested in using or replicating the current study can take advantage of the work that has been done.