Learn more: PMC Disclaimer | PMC Copyright Notice
Assessment and evaluation is not a gut feeling: integrating assessment and evaluation into library operations*†
The authors love assessment and evaluation, and we believe it should be integrated into all aspects of library programming. There. It's been said. Reaching this point in our relationship with assessment and evaluation has definitely had its ups and downs, and we still struggle with it. So why speak out about this now? The current economic environment makes it more important than ever for libraries to showcase their contributions to their institutions. Several hospital libraries have been closed in the National Network of Libraries of Medicine, MidContinental Region (NN/LM MCR), in the last few years. The Veterans Administration Hospitals, anticipating boomer librarians' retirements are looking at large numbers of vacancies with the possibility of no replacements. While academic health sciences librarians are not anticipating their libraries being closed, they are fighting for replacements for departing and retiring librarians and for budgets to offer services to support more complex needs such as data management and curation, systematic reviews, and provision of clinical information at the point of care. Health care reform is requiring more accountability. When assessment and evaluation are pervasive, librarians have the evidence to effectively communicate how each part of the library contributes to the success of the institution.
The authors believe in the value of librarians and libraries and that they contribute to the mission of the hospitals, health centers, universities, and organizations they serve. Librarians have stories to tell, but the data currently collected tend to focus on isolated factors that do not tell the whole story. Librarians are doing ourselves a disservice by not providing data that show how we support meaningful use and demonstrate how library resources and services contribute to the institutional mission. Health sciences librarians must ask: What do librarians really contribute to our institutions? Are patients receiving better care because of what librarians are doing? Which activities best support our users? How does our training enhance lifelong learning skills? How do librarians contribute to the research process? What infrastructure is needed for these activities? Once these questions have been asked, librarians must decide what metrics will help us address them.
Several terms related to this discussion require definition. Assessment and evaluation are often used interchangeably but they are, in fact, different activities. By assessment, the authors mean the process of continually reviewing work in progress. By evaluation, we mean review done at a point in time allowing the evaluator to make judgments or draw conclusions about the impact and achievement of the desired outcomes of an action. Value is a term frequently used when discussing a library's contribution to the parent organization. The word value implies merit, importance, or worth. In the context of evaluation, we are also speaking of value-added, value that exceeds expectations. And finally, there are metrics, a set of accepted or agreed upon measures. Many things can be measured, but metrics are relevant and significant to the question at hand. Metrics are usually, but not always, quantifiable. The assessment and evaluation practices of two organizations, discussed below, illustrate the application of these terms and the value that can be derived when the process is pervasive through the organization.
National Network of Libraries of Medicine, MidContinental RegionIn 2001, the National Library of Medicine (NLM) awarded the NN/LM contract to the Spencer S. Eccles Health Sciences Library at the University of Utah. With this award came a new model of service to the region, one that incorporated partnering with Resource Libraries (leading academic libraries) in each state and establishing staff at each of the Resource Libraries to carry out the NN/LM five-year contract.
This new program was challenged to not only coordinate a distributed staff, but to prove to the NLM and to members of the region that this new model would work. Staff wanted to show not that we were better than a centralized program, but that we could be as effective in addressing the needs of the region's constituents. This desire to show impact was written into the proposal. We realized that we were at a critical moment in our history and that, since we were at the beginning of a first-time contract, we had the opportunity to collect measures that could be later used to show improvement. Libraries can establish any day as their first day and collect baseline data to start measuring their effectiveness, but measures must be taken, or there is no way to chart progress.
To learn who our members are, we have gathered information on them once every five years: information about their staffing, place and involvement in their parent organizations, budgets, collections, teaching, technology, outreach, and use of various NLM resources, as well as programs or services that have gained importance among libraries since the last assessment. In addition to providing details on our members, this assessment also alerts us to the ways in which their library environments are changing. We have a core set of topics (e.g., technology), but within each topic, the measures may change over time. For example while connectivity was measured by bandwidth in 2003, today the metric is whether institutional policy blocks access to online services and resources such as social media.
The NN/LM MCR program focuses on six areas of outreach: assessment and evaluation, library advocacy, health information literacy, education, technology, and member services. Each of these areas develops an annual plan to reach the five-year goals established by NLM. We regularly assess whether we carried out the activities that we set for ourselves and evaluate to what extent we met the indicators for project outcomes. For example, in the last few years, the technology program has judged its effectiveness by the number of members who adopted a technology that was promoted or taught. Our health information literacy initiatives have been measured by the feedback received from collaborating organizations who recount what we did to improve their access to information. These stories, which relate the impact we have on the region, are reported back to NLM so that they understand the value we are adding to the region. Based on our experience, we know that our metrics provide information on our impact on our user population and that by making assessment and evaluation pervasive, the organization and all of its programs provide a holistic view of what we are doing to achieve the NN/LM mission and goals and the progress in achieving them over the life of the contract.
Bernard Becker Medical Library
At Washington University in St. Louis, the Bernard Becker Medical Library's paper forms have given way to databases where more complex metrics can be recorded and a variety of data collected and manipulated. To collect information that truly documents the use and value of the library, the library revised the Library Statistics Application created by the University of New Mexico 1. More of the library's collections are electronic, making traditional gate count, circulation, and shelving statistics irrelevant. Library Strategic Directions 2012–2015, the library's three-year strategic plan, addresses each of the four missions of the School of Medicine 2. It outlines eighteen indicators for evaluating progress in seven strategic directions. The Becker statistics software provides a way to demonstrate the breadth and depth of services provided by the staff. Many of the metrics needed to demonstrate progress on the strategic plan are captured in the statistics database. Staff know what services we are providing, how often, and for whom. We capture names and email addresses of those we work with and anecdotal or qualitative notes about the interaction. These data can be quickly analyzed and pulled into Microsoft Access reports that summarize how both the librarians and the paraprofessionals contribute to the library's strategic plan and the School of Medicine's mission. The analysis is included in the library's annual report that accompanies our budget request to administration. Data collection began in late 2010 so we cannot yet demonstrate whether providing this information to the administration has had an impact. However, the consistency and detail of the collected data have allowed us to illustrate that the library's services are evolving and growing and that the library is involved and invested in the education, research, patient care, and outreach mission of the School of Medicine.
Libraries have traditionally collected data about resources and services and have primarily used it to compare themselves with other libraries. However, these data do not provide the whole story of the impact the library has on its own institution nor how the library contributes to the institutional mission. Think of the parable from India of the elephant and the blind men. By touching a single part of the elephant, each blind man understands the elephant as the part that he touched. Most libraries conduct evaluation the same way. They take a single service (clinical librarianship) or a single department (systems) and evaluate productivity, fulfillment of need, or user satisfaction. From this evaluation, all that can be known about the library is the trunk or leg or tusk. In her article, “The Postmodern Library in an Age of Assessment,” Ray describes the modern library as being characterized by “predictability, certainty, control, absolute truths, and order” 3. Health sciences libraries are no longer in that modern age. Today is the age of the postmodern library, characterized by “subjectivity and multiple realities; plurality and diversity” 3. The results of evaluation efforts need to reflect this complexity, so that a multidimensional story can be related to our stakeholders, and librarians can better understand the complexity of our own libraries.
Accrediting organizations and government entities expect higher education and health care to demonstrate how institutions meet their missions and improve the quality of patient care and the educational experience. Libraries are having similar demands placed on them. Librarians need to be leaders in demonstrating how to effectively gather and use information to illustrate and promote our value. To become leaders, health sciences librarians need to integrate assessment and evaluation into our organizations, collect data for metrics that define our value, and report the evidence of our value to our stakeholders.
The authors look forward to the time envisioned by Ray when assessment and evaluation “is assimilated into the daily work of each staff member. Assessment is no longer viewed as an externally imposed chore or a detour from normal routines. It becomes a self-organizing principle and helps create an organizational climate that encourages inquiry, exploration and reflection” 3.
Footnotes
*.This project has been funded in whole or in part with federal funds from the Department of Health and Human Services, National Institutes of Health, National Library of Medicine, under contract no. HHS-N-276-2011-00006-C with the University of Utah Spencer S. Eccles Health Sciences Library.
†.Based on a presentation at MLA '11, the 111th Annual Meeting of the Medical Library Association; Minneapolis, MN; May 2011.