Characteristics of project management at institutions sponsoring National Library of Medicine MedlinePlus Go Local.

Objectives: Through interviews with the National Library of Medicine’s MedlinePlus Go Local collaborators, an evaluation team sought to identify process characteristics that are critical for long-term sustaina-bilityofGoLocalprojectsandtodescribetheimpactthatGoLocalprojectshaveonsponsoringinstitutions. Methods: Go Local project coordinators (n 5 44) at 31 sponsor institutions participated in semi-structured interviews about their experiences developing and maintaining Go Local sites. Interviews were summarized, checked for accuracy by the participating librarians, and analyzed using a general inductive methodology. Results: Institutional factors that support Go Local projects were identified through the interviews, as well as strategies for staffing and partnerships with external organizations. Positive outcomes for sponsoring institutions also were identified. Conclusions: The findings may influence the National Library of Medicine team’s decisions about improvements to its Go Local system and the support it provides to sponsoring institutions. The findings may benefit current sponsoring institutions as well as those considering or planning a Go Local project.


INTRODUCTION
The National Library of Medicine's (NLM's) Med-linePlus Go Local [1] provides statewide or regional databases of health-related services linked to health topics in MedlinePlus, NLM's consumer health website. Links between MedlinePlus and Go Local allow users to move easily between researching a health topic and searching for a health service related to the topic. For instance, a user in Cook County, Illinois, can look up information about arthritis on MedlinePlus, then click the Go Local link on the topic page to find rheumatologists practicing in or near that county. Alternatively, a user can search Go Local for an area acupuncturist, then use the MedlinePlus link to locate information about acupuncture treatment.
NLM's motivation for starting Go Local was to provide a means for the public to find relevant health services from the health content on MedlinePlus. The long-term goal of MedlinePlus Go Local is to improve access to health services by providing a wellorganized, sustainable, up-to-date, and useful collection of health services that serves an entire geographic area, as defined by the sponsoring institution. Staff at sponsoring institutions in the United States compile and manage a collection of local health services for their state or regions, with most using a central computer system managed by NLM. Most sponsoring institutions are health sciences or hospital libraries, but several projects have been managed by other organizations such as a community-based crisis N An institutional record of community service or outreach and a director who was supportive of the project were important factors in the progress and sustainability of Go Local projects.
N Go Local projects brought recognition to some sponsoring institutions from their parent institutions or their communities and provided opportunities to establish better relations with other libraries and institutions.
N Go Local projects ran more smoothly when a person, even a temporary hire, with dedicated time for the project was in charge during the initial building phase. Volunteer assistance has been difficult to motivate and sustain. intervention agency and a university-based rural health research center. Go Local is not available in all states.
Sponsoring sites dedicate a tremendous amount of resources-specifically staff time-to Go Local projects on an ongoing basis. The initial effort of locating and entering health services information into the database is extremely time consuming, and the National Network of Libraries of Medicine's (NN/LM's) Regional Medical Libraries provide nominal one-time funding to sites. However, the process of auditing and updating records, which are checked regularly, is also labor intensive and requires an ongoing commitment from the sponsoring institutions.
The first Go Local site was NC Health Info, started in 2001 when NLM funded a three-year pilot with the Health Sciences Library and the School of Information and Library Science at the University of North Carolina (UNC), Chapel Hill. NC Health Info launched in early 2003 [2][3][4].
After the success of the pilot site, NLM found early adopters who wanted to start a Go Local service for their own state or region. Because the UNC pilot project demonstrated the significant expense of creating and maintaining a software system to manage Go Local and NLM could centralize the system and provide it to all sites, NLM staff decided to take the cost-effective route of building a centralized system that could be used for any Go Local project. The team relied extensively on lessons learned from the UNC pilot project and experience with MedlinePlus's management software [5,6] to build the NLM-hosted Go Local system, released in May 2004. To build coverage of states and regions, NLM, through the NN/LM Regional Medical Libraries, began providing seed funding for developing Go Local sites with start-up awards of up to $25,000.
Since the NC Health Info release in 2003, 32 additional Go Local sites have become part of the program and serve areas that include nearly 45% of the US population. Six new sites are under development. Currently, all but 2 Go Local sites, North Carolina and South Carolina, use the NLM-hosted system. The NLM-hosted Go Local sites had an average of 348,000 page views per quarter during 2008.
In 2007, the NLM organization that primarily administers Go Local and MedlinePlus-the Public Services Division, Library Operations-secured a National Institutes of Health (NIH) evaluation award to investigate the impact of Go Local projects on the sponsoring sites. An evaluation team including two librarians from the NLM Go Local staff, along with an evaluation consultant, conducted interviews with Go Local collaborators to hear, in detail, about their experiences developing and maintaining Go Local databases. The primary goal was to gather information about the process characteristics believed to be critical for the long-term sustainability of Go Local projects and to understand the effect that Go Local projects have on sponsoring institutions. NLM's goals for Go Local projects included (a) efficient operations that apply institutional resources only to the extent necessary to maintain a reliable, credible website; (b) increased visibility and reach of the sponsoring institutions within and outside their parent institutions; (c) formation of positive organizational partnerships in communities that improve the service; and (d) sustained service over time. The evaluation team hoped to learn strategies used by sponsoring institutions to pursue these process goals. Conversely, the evaluation team wanted to develop more awareness of potential threats to these goals that might affect sustainability of the sponsoring institutions.

CURRENT STUDY AND METHODS
The evaluation team chose a qualitative methodology, interviewing, to collect detailed descriptions of how Go Local teams at sponsoring institutions implemented their projects. The independent evaluation consultant on the team had approximately six years of experience working with NLM-funded consumer health projects, including an evaluation of the Go Local pilot developed at UNC. Both NLM librarians had extensive experience with Go Local: one started working on the project during the feasibility study at UNC, and the other joined the NLM Go Local team during the project's expansion in 2004.
The evaluation consultant conducted all interviews with project collaborators (who had a variety of titles but will be referred to as ''project coordinators'' throughout this article). She also directed the qualitative analysis process and served as primary author of project reports. The NLM librarians gained consent and cooperation from Go Local project coordinators, consulted on the interview guide, assisted with analysis of the qualitative data, and coauthored reports. A third librarian, from NLM's NN/LM National Network Office, assisted with coding the evaluation data during the analysis phase of this project.

Participants
Interviews were conducted with project coordinators and, sometimes, other Go Local team members at each of thirty-one sites{ that had launched or were under development in August 2007. Twenty-four sites had been launched, and seven were under development at the time of their site interviews. A total of forty-four Go Local team members were interviewed.

Interviews
Interviews were conducted by telephone between November 2007 and May 2008. The interviews were semi-structured: The interviewer followed a standard set of questions (Appendix, online only) but had latitude to explore topics in more depth if appropri-{ There was a thirty-second site at the time of this project: the Tribal Connections Four Corners (TC4C) Go Local. However, this project is considerably more complex, covering a geographic region spanning four states, so TC4C was not included in this analysis. Olney et al. ate. The questions, organized under nine basic topics, helped the evaluation team explore how the sponsoring institutions pursued Go Local process goals, effects of the project on the sponsoring institution, and potential threats to the project's sustainability ( Table 1).
The evaluation team developed an interview guide, reviewed by others at NLM who were associated with the Go Local project, and piloted it with 3 Go Local sites. Minor revisions were made to the questionnaire. The pilot site data were included in the overall study. Most interviews lasted 90-120 minutes.
The evaluator typed extensive notes during the telephone interviews, creating an abridged transcript for each interview. She then wrote an interview summary for each site and sent it to the interviewee to confirm accuracy or make corrections. To build trust and allow for an honest exchange, interviewees were permitted to remove information they wanted to keep off the record or request rewrites for passages that they felt did not adequately represent their views. Interview summaries were three to eight pages long.

Analysis
Data were analyzed using a general inductive approach described by Thomas [7] designed specifically for evaluation projects. This approach is similar to the analytic processes of grounded theory [8] and the pattern coding methods described by Miles and Huberman [9] but has been modified to focus the inquiry on qualitative content directly relevant to the evaluation objectives. The evaluation consultant read through the first sixteen interviews and developed codes for themes related to characteristics of the sponsoring institutions, operational strategies, partnerships, positive outcomes for sponsoring institutions, and threats to sustainability. She then worked with three other coders (the two evaluation team members and the NN/LM National Network Office librarian) to refine the list of codes and their definitions. Then, all summaries were thoroughly coded by the evaluation consultant and one other coder on the team. Coders discussed each summary to resolve discrepancies in coding.
Using a data-display technique described in Miles and Huberman [9], a matrix was constructed to graphically show the absence and presence of themes across sites. This allowed the team to identify the most prevalent themes. They decided that themes appearing in sixteen summaries (more than half) represented a relatively common element of Go Local projects.
To confirm that themes were related to project effectiveness, the team asked four NLM staff members who were responsible for managing the Go Local project to independently list criteria of sites that were ''strong'' (e.g., stable and thriving). These staff members had worked closely with Go Local sites and had been involved in developing individual sites, providing training, guiding sites as they launched, testing and monitoring site content, and supporting their ongoing maintenance. The staff members identified the following characteristics: & The project had a plan for regularly auditing the accuracy of database records. & Records were being audited on schedule. & The website and project were being promoted regularly by institutional staff. & Projects under development showed momentum in adding records to the database, and the staff who input records understood how to make appropriate indexing choices.
The same four NLM staff members then independently listed sites they believed fit their criteria. If a site was listed by at least three of them, it was included in the list of ''strong'' sites for the analysis. Using this process, eleven sites were identified as strong sites and were compared to the total group to see if the themes occurred at a higher rate in this subgroup. Table 2 presents the themes that emerged in at least sixteen interviews, which the evaluation team set as the standard for a strong theme. The table also shows the number of times the themes emerged in the ''strong site'' interviews.

Institutional factors
Commitment of sponsoring libraries or organizations to consumer health information or community A committed project coordinator was critical: Many project coordinators said that, although they knew Go Local projects would be a lot of work, they still were surprised at the amount of time the project required. Some worked almost 100% of their time to get the site ready for launch. In a number of cases, project coordinators left sponsoring institutions for jobs in different institutions but continued to be involved in the Go Local site for their states or regions. In some cases, their assistance was temporary until they could be replaced. For example, one former sponsoring institution withdrew from the Go Local project when library staff was downsized and the Go Local team members found jobs elsewhere or retired. Because she did not want the project to fail, one team member acquired permission from her new director to provide a minimal level of maintenance to Go Local-''life support,'' as she called it-until NLM could find a new sponsoring institution. In another case, a project coordinator, who had a co-coordinator at another institution in the state, chose to remain with the project indefinitely after changing jobs. A deep commitment to the project could also be seen in team members who assumed responsibility for the Go Local project when the original project coordinators left an organization and were not replaced. Dwindling staff meant increased workload for these project coordinators, but they maintained their sites as much as they could and sought resolutions to their staffing problems, such as building a partnership with another organization. Support from library director or head of an organization. More than half of the project coordinators said the active support of library directors was critical to a Go Local project's success. Library directors provided support in a variety of ways, such as committing resources to Go Local when NLM monies ran out, providing funds to hire temporary employees both during development and maintenance phases, assigning Go Local work to library staff, reorganizing the staff to support the Go Local project, or working on records themselves.

Staffing of Go Local projects
Access to temporary help during development stage. Most of the sites hired temporary workers to build the Go Local database, often using the NLM Go Local start-up awards to pay for temporary help. The six sites that did not hire temporary workers spread the work among existing staff. One hiring strategy was identified as successful by at least two project coordinators: One temporary assistant was hired, rather than several workers who worked a couple of hours a week on the project. Others helped with record input, but the temporary assistant oversaw record entry and maintained a consistency in the level of detail and style of each record. One project coordinator added that having the assistant work one day a week at the library made the database development run more smoothly, because the assistant could talk face-to-face with others working on records. Both project coordinators knew they were fortunate to find temporary assistants with the maturity and skills to lead the database development.
Temporary assistants with some relevant background, such as in libraries or health care. When possible, project coordinators hired temporary assistants with some type of relevant background, such as graduate students from library and information science graduate programs, retired librarians or support staff, or partially employed or unemployed librarians. Those who worked with a variety of Olney et al.
temporary workers found that those with backgrounds that had exposed them to medical terminology were more comfortable matching vocabulary to services. Finding temporary assistants with any type of relevant background was not possible for some Go Local teams.
Assistance from staff beyond the Go Local team.
Almost two-thirds of the sites pulled in support from staff who were not on the core Go Local team, usually during the database development phase. Often library staff and student assistants were assigned to help with developing the database, or librarians contributed their time to work with records in topic areas of particular interest to them. At one sponsoring institution, the project coordinator said that the staff had ''Go Local fever'' and that almost every staff person wanted to make some type of contribution to the project. Most of the extra staff support came during the development stage, then extra support was minimized once the project was launched.
Outreach librarian or outreach unit. Outreach was an important part of any Go Local project and often began in the development stage. Not only did project teams need to promote Go Local to users, they also had to reach out to organizations to maintain information for Go Local records. If institutions had outreach librarians, promotion of Go Local was added to their promotional activities. Outreach efforts allowed Go Local teams to secure health information (e.g., from public health organizations) or develop partnerships with organizations that could help promote the database or provide funding (e.g., library associations). At sponsoring institutions where the outreach librarians were also responsible for the technical part of the project, they did not do much outreach until after launch.
Division of responsibility: project management, database management, and outreach. The amount of time needed for promotion was significant, so outreach and technical responsibilities were sometimes divided between two staff members, particularly if the sponsoring institutions had an outreach person or librarian. Sometimes two people divided responsibilities between technical development of the database and project management, which included outreach to organizations (e.g., to get health service information or funding), promotion, and fund-raising. Often, the project manager had worked in the state or region long enough to form relationships with colleagues in other libraries or organizations.
External support National Library of Medicine infrastructure. The most frequently mentioned form of external support was NLM's Go Local infrastructure and support services. Several people said they understood how much time NLM saved them by providing a centralized system to manage the Go Local databases, and a number said the NLM Go Local team frequently made system improvements based on site team feedback. Almost every project coordinator also took advantage of the NLM support services, including the Go Local Extranet, the email discussion list, quarterly teleconferences, and face-to-face meetings at the annual meeting of the Medical Library Association (MLA). Many project coordinators mentioned how quickly the NLM Go Local team responded to questions. NLM's efforts seemed to have created a networking atmosphere among sponsoring site teams, because several project coordinators contacted each other directly for advice when starting projects.
Partnerships with another library: health sciences or public. More than two-thirds of the project coordinators developed partnerships with other libraries, usually another health sciences or public library. Thirteen project coordinators said they developed relationships with library associations, which sometimes provided funding, played an advisory role, helped the Go Local team network with association members, or helped promote the site. At least ten Go Local teams have forged partnerships with state libraries, which connected them to public libraries or provided funding.
Partnerships with nonlibrary groups: public health departments and centers for disability services.
More than half of the sites have built relationships with nonlibrary organizations, such as public health departments and centers for disability services. These organizations often provided health service information for the database. For example, the dean of a school of allied health shared a database of rural health services compiled by students during their preceptorships. The partnership was mutually beneficial: The Go Local team received good health service information, and the school had a more searchable database. Representatives of nonlibrary partners also served on advisory committees, which about onethird of the sites used, and the representatives mostly provided feedback about the site and helped with promotion.
United Way 2-1-1. A link between the United Way's 2-1-1 service, which help consumers find community social services, and Go Local would seem to be intuitive, and seventeen project coordinators said their teams attempted to form relationships with 2-1-1 call centers. Of those seventeen sites, four reported successful relationships with 2-1-1 agencies, which shared resources and used Go Local for referrals. In one case, a call center director served on the Go Local advisory board. Those with failed attempts at building partnerships with 2-1-1 agencies said 2-1-1 staff members did not think Go Local could add to their services and, Go Local project management sometimes, perceived Go Local as a competitor for the same funding sources. High staff turnover at some 2-1-1 agencies occasionally posed a problem.
The project coordinators who did not attempt to form partnerships with 2-1-1 agencies cited various reasons. Program coordinators from newer Go Local projects knew of the mixed success of their colleagues. Furthermore, in most states, 2-1-1 services were organized locally, not statewide, so project coordinators would have had to build relationships with multiple 2-1-1 agencies. Some areas had little or no 2-1-1 coverage. Also, some project coordinators suspected the 2-1-1 records were out of date. Probably the biggest barrier was the difficulty of sharing records between 2-1-1 and Go Local systems, due to vocabulary and database structure issues, so a partnership offered no technological advantage to either group.
Volunteers. Recruitment of volunteers seems like an obvious strategy for distributing the workload of record upkeep. In fact, project coordinators from 13 sites did report an effort to incorporate volunteer help into their projects, and all rated their efforts as minimally successful at best. Some sites trained groups of volunteers (usually hospital or public librarians) but usually retained very few over the long term. For example, one project coordinator recruited public library directors to confirm database records from services in their towns. To facilitate this effort, she printed out records from areas in the state, then gave the printouts to public library directors and asked them to confirm the information. In spite of training, incentives (a $50 gift certificate), and monthly calls, the project coordinator struggled to get minimal participation from the volunteers.
In a unique effort, a Go Local team is negotiating with the director of a graduate library program to train library school graduate students to audit Go Local records. If the plan works as expected, the students will audit records as part of their fieldwork requirement.

Positive outcomes
More recognition with the outside community. About two-thirds of the project coordinators said Go Local projects brought more community recognition to the sponsoring institutions. It was not uncommon for Go Local launches to attract television coverage, and a number of project coordinators said they had been asked to present or exhibit to attendees of library association meetings, community programs offered by public libraries, and community health fairs and to family practice residents.
More recognition in their institutions. About half of the project coordinators thought Go Local increased their organization's recognition in their institutions. In several cases, Go Local captured the attention of administrators who had a mission to promote community relations. For instance, one project coor-dinator said Go Local team members had traveled with their university's alumni association on statewide promotional tours. At least two project coordinators said faculty members talked about using Go Local to locate internship sites for their students. In yet another case, a project coordinator said her library was asked to partner with their university's school of social work on a high-profile community outreach program that incorporated promotion of NLM resources, including Go Local.
Other benefits. More than half of the sponsoring institutions listed a variety of other positive outcomes from their Go Local project. Several said that Go Local helped them extend their outreach to community members with health issues, and a number said their organizational staff used Go Local as a reference tool to help users. Some project coordinators thought the Go Local project itself was a good team-building project for their libraries. One said it allowed professional and paraprofessional staff in their library to work together across unit lines. Several project coordinators said the Go Local databases are good resources for identification of medically underserved areas and, in at least one case, for advocacy with state legislators.

Sustainability
Almost all of the themes discussed in this paper were found at a higher rate among the strong sites, suggesting they contribute to the stability of a project. Only the use of the NLM infrastructure and support services did not distinguish the strong sites from the others, because almost all sites took advantage of this support.
Project coordinators were asked to describe potential threats to their projects' sustainability, and for the most part, they were fairly optimistic about the continuation of their Go Local sites. Seven interviewees acknowledged that resignation or retirement of their directors, who were strong supporters of the project, could threaten sustainability if the incoming director brought new priorities that replaced Go Local. Small library staff, downsizing, lack of administrative support for the project, reassignment of Go Local team members to different projects, and loss of the project coordinator were other threats mentioned by approximately the same number of project coordinators. Two project coordinators said that, if they themselves left, it might be hard to find another staff member with time to learn how to manage a Go Local project.
A number of Go Local projects have faced these circumstances, but, to date, only one has been taken offline. However, staff downsizing has led to a struggle for several project coordinators, who must juggle the maintenance of the database with other added responsibilities. In some cases, pursuit of funded projects takes precedence over Go Local maintenance because the organization relies primarily or solely on soft money. Yet, project coordinators do Olney et al. not want to see their sites fail, so most of them do what they can to sustain the sites until other arrangements can be made.

DISCUSSION
The primary goals of this evaluation project were to (a) identify characteristics of Go Local projects critical to their long-term sustainability; (b) describe the effect that Go Local projects have on sponsoring institutions; (c) describe the strategies that sponsoring institutions use to pursue the process goals of efficient project management, increased visibility and reach, positive organizational partnerships in the community, and sustained service; and (d) identify potential threats to Go Local projects. The characteristics found in this evaluation study that seemed most critical to long-term sustainability were the high level of dedication of at least one person on the staff who took responsibility for the project and the full support of the library director. It also helped if the sponsoring organization had a commitment to consumer health or community outreach, because the organizational mission supported the work. Many interviewees expressed optimism that their Go Local projects would be sustained because the projects had the full support of their directors. However, many believed the loss of a committed director or shifting priorities in the parent organization that would force a director to re-prioritize programs could threaten the survival of their Go Local projects.
Most project coordinators said their Go Local project had limited negative effect on their sponsoring institutions because they hired temporary assistants during the development phase and did the bulk of the maintenance work themselves once NLM's funding was expended. In a few cases, organizational staff was restructured to support the Go Local project. Some described positive organizational outcomes, such as more recognition for their organizations or enhanced ability to reach out to consumers. Both outreach and development of partnerships with external organizations were stronger if more than one person was involved with the Go Local project. It was difficult for individuals who manage the technical side of Go Local to also do outreach, particularly during the development stage. If the individuals were also the outreach librarian, for instance, they usually did outreach after the development stage.
This study had some obvious limitations. The evaluation team chose to focus on characteristics, decisions, and behaviors of the sponsoring institution's staff who manage the projects. The effect of demographic characteristics, such as the organization's size or size and nature of the population served (i.e., urban versus rural), were not explored. It is possible that these demographic variables contributed to the stability of a Go Local project and could be a direction for further research.
Also, discussion about ongoing funding from NLM (i.e., in addition to the initial start-up funds) was minimized. The challenges of ongoing funding of Go Local were a topic of concern among Go Local collaborators, and some project coordinators mentioned that ongoing or additional funding from NLM would help them continue to build the databases (rather than simply sustain them) or increase the pace of record auditing. However, all project coordinators knew when they wrote their proposals that NLM's financial support was limited to start-up funds, and most did not expect these circumstances to change. A final limitation that should be noted is that the participants knew the interviewer was hired by NLM, and this factor might have affected the frankness of the conversations.
The evaluation project also showed that the project coordinators relied heavily on-and were quite satisfied with-the support they received from NLM. This finding was consistent with results of a survey of project coordinators conducted by Hogan, MacCall, and Vucovich, which demonstrated high levels of satisfaction with NLM's support among Go Local project collaborators [10].

OUTCOMES
Given the themes related to project stability that emerged in this project, the NLM librarians on the evaluation team said they believed the Go Local project guidelines were on the mark for helping potential sponsoring institutions plan a Go Local project. The data did lead them to conclude that they must reemphasize that prospective project coordinators thoroughly explore what level of support they can expect from their library directors when the start-up funds are depleted. This evaluation project also helped them identify some emerging issues to address.

System efficiencies
During the interviews, many project coordinators described frustrations with some specific Go Local system operations. To further explore ways to address these frustrations, the evaluation team scheduled two discussions at MLA '08. The interviews and subsequent discussions have allowed the NLM Go Local staff to identify and prioritize operational changes that will enhance system efficiency, such as streamlining record creation, allowing managers to bypass secondary approvals, and improving functions for global updates and imports, all of which would save time for Go Local staff.

Program continuity
The evaluation team discovered that not all program coordinators were prepared to pass their Go Local responsibilities on to another staff member, should they leave their positions. The NLM Go Local team is considering developing guidelines to help project coordinators document their responsibilities to assist program continuity. They also will plan a more formal approach to orientation of new project coordinators.

Go Local project management
Volunteers Quite a few project coordinators talked about the potential benefits of working with volunteers who lived in their states. Go Local project coordinators have a limited geographic region that they know well, bounded by where they work and live, and they must rely on printed records and online websites for areas outside those boundaries. They believed that a network of volunteers throughout their state could provide more accurate, detailed, and comprehensive information than the library staff alone could compile. Yet no Go Local team has found a highly successful way to motivate Go Local volunteers to become or stay involved. NLM might consider funding pilot projects that are focused on developing successful Go Local volunteer training programs.

CONCLUSION
The evaluation project has provided insight into the experience of Go Local project coordinators and their teams. It has demonstrated some of the institutional factors and management strategies that contribute to the stability of Go Local projects, which require ongoing commitment of staff time. With this information, both NLM and project coordinators are more prepared to improve the process in priority areas and plan for strengthening the Go Local project in the future.