Content Selection and Editorial Workflow

To create timely and high-quality weekly AHRQ PSNet issues, we needed a system that was fluid and dependable, could function as a content management tool, and would alert different members of the editorial team when their tasks were ready for assignment. Our technical contractor helped develop an online authoring tool that provided the necessary structure and organization to help publish our new issue every Wednesday.

Each week, our library scientist and editorial team identify potentially relevant content via systematic searches of bibliographic databases (e.g., PubMed) and also several other clinical, health care administration, business, legal, and lay press publications (e.g., newspapers and magazines). We also closely follow industry and consumer dialogue on patient safety issues via blogs, LISTSERV™ applications, and Web-facilitated news and site update alerts. Finally, our managing editor is able to anticipate inclusion of important resources in a timely manner through media access to upcoming, embargoed publications.

One of our first editorial decisions during site development focused on the number of new resources we could reliably add in a given week. We decided to choose quality over quantity, opting to provide users with editorial input into new content, rather than simply including all relevant content and potentially overwhelming users. After agreeing on a starting set of resources prior to launch (with larger input from our expert Editorial Board), we targeted inclusion of 20 to 25 new resources each week (~1,000/year) spanning journal articles, newspaper stories, conference proceedings, toolkits, and reports.

Our editorial team spotlights certain items each week by accompanying the resource with an annotated summary. These summaries (approximately 100 to 150 words) aim to highlight important information from the resource and lead users to related content, glossary items, or AHRQ WebM&M commentaries via hyperlinks. Thus, while reading an annotated summary, a user is also directed to other relevant literature elsewhere on AHRQ PSNet, a feature that enhances the user experience (Figure 1). The remaining resources receive shorter summaries with the same linking principles to create similar depth to each resource description. Evaluation of past user behavior (discussed in more detail in the Results section) has allowed us to tailor our editorial decisionmaking over time. During the last steps of our process, the “What’s New” home page is designed, with resources chosen, prioritized, and highlighted for publication.

Figure 1

AHRQ PSNet sample annotation.

One recurring challenge of our content selection process comes as we try to determine whether a resource is about “safety” (our mandated scope) or “quality” (generally, outside our scope). The distinction between “patient safety” and “health care quality” is by no means clear and is, to some extent, purely academic.⁷ Although our charge is to focus on patient safety, we do not want to inappropriately exclude quality-focused resources that are also relevant to a safety-oriented audience. For example, a study on the use of information technology (e.g., computerized provider order entry, CPOE) that improved the quality of care delivered to patients with diabetes might be excluded as quality-focused. On the other hand, a similar study demonstrating the role CPOE played in improving beta-blocker use in the perioperative setting might warrant inclusion, given that perioperative beta-blocker use was identified as an important patient safety intervention in an AHRQ technical review (e.g., that defined one aspect of safety practices as cross-cutting).⁸ The balance here remains challenging and generally defies fixed rules for inclusion and exclusion.

The broader issue involves defining the “market” for patient safety information. Does every black box warning about a medication or device require inclusion? Does an article focused on quality but relating to safety published in a marquee journal trump a safety-specific article in a lower impact journal? These sorts of discussions are resolved through consensus and with an eye toward past user feedback, while keeping in mind our overarching mission: to err on the side of high-quality resources, rather than aiming for an inclusiveness that would likely generate an overwhelming amount of content and a poor user experience.

Site Attributes

During site development, we identified the chief site attributes as:

• Timely sharing of new information.
• An attractive, usable, and intuitive user interface.
• A customized and searchable set of resources using a taxonomy that offered multiple axes.
• A balance between dynamic content (e.g., literature, meetings, and news) and resource content (e.g., toolkits, conference proceedings, and legislation).

Below, we highlight a few specific features that have been popular with our users and demonstrate the efforts to create a particular user experience on AHRQ PSNet.

What’s New and AHRQ PSNet Newsletter

We designed an interface on the site to highlight new content—i.e., “What’s New”—and an active way to alert users of the new content: the electronic newsletter. With 20 to 25 carefully chosen resources each week, we still felt the number could potentially overwhelm users on the home page. Instead, we produced a dynamic left side of the home page for “What’s New” and a static right side of the home page for existing content (Figure 2).

Figure 2

AHRQ PSNet home page.

In “What’s New,” we select the top 10 to 12 resources each week, organized by resource type (i.e., journal article, newspaper/magazine article, Web resource), and prioritize them based on desired connection between the two sites.

To cue users to explore new content, we provide an opportunity to receive an AHRQ PSNet newsletter that is electronically delivered to subscribers. The email displays the “What’s New” content and allows users to link directly to the individual resources on AHRQ PSNet. We believe this feature is vital for keeping our users easily updated, a primary objective during our site development.

My PSNet

The “My PSNet” option allows users to view the latest resources available in their self-designated areas of interest. When a new resource matching their specifications is added to AHRQ PSNet, they receive e-mail alerts up to once weekly. The process begins by walking users through a series of check boxes to highlight their interests—e.g., choosing a safety target, an approach to improving safety, a setting of care, a clinical area, a target audience, and an error type. Each of these categories drills down to more specific areas, so each user can customize preferences as broadly as “All Approaches to Improving Safety,” or simply “Teamwork Training.” We believed that by helping users define their areas of interest, matched with our custom developed taxonomy (described below), the user experience would be optimized.

Classics

Given the volume of content on AHRQ PSNet, we wanted to highlight enduring and influential resources in patient safety—i.e., our “Classics.” After agreeing on an initial list of classics at site launch (with input from our Editorial Board), our editorial team reviews resources every 6 months to designate new “Classics.” Typically, we start with a list of potential resources—all of our selected “key” articles (i.e., those with longer annotation summaries), those highlighted on the “What’s New” home page, and the most frequently viewed and cited resources. The editorial team convenes, chooses a target of approximately 50 new classics per year, and then seeks input from the Editorial Board and Advisory Panel. Once final selections are made, these resources receive the “Classic” designation, increasing its weight and importance in our searching algorithms. We believe this feature provides users, particularly those new to the field, with an easy method to identify landmark resources. A few times a year, we will also designate a particularly noteworthy new resource as an “Instant Classic.”

Glossary

Building on the success of the AHRQ WebM&M glossary, we reproduced and expanded the glossary on AHRQ PSNet. The glossary terms have grown in number and depth, as many contain links (similar to our annotations) and important references. Given the breadth of the field, a comprehensive glossary helps explain commonly used terms (e.g., “safety culture”) and activities (e.g., root cause analysis) in patient safety. Adding or modifying glossary terms is part of the editorial workflow; new terms are identified or raised by editorial members, often while writing weekly annotations for new resources. AHRQ PSNet and AHRQ WebM&M glossaries are shared, and searching for a phrase on AHRQ PSNet, such as “safety culture,” will provide both the glossary term and the available resources matching the search term.

Taxonomy

A good user experience on AHRQ PSNet requires sensitive and specific methods for visitors to locate resources. A simple text search (e.g., “culture”) across the site’s resources would be inadequately specific, whereas an unstructured keyword list would quickly become unmanageable due to size and internally redundant. Therefore, we designed a structured categorization of descriptive terms—i.e., a taxonomy—to label resources on AHRQ PSNet. The taxonomy was designed by consensus and iterative review by our editorial team. Taxonomies composed by such expert groups tend to be large; they balance high specificity when describing complex domains at the cost of complexity and decreased usability. We tried to minimize these limitations by carefully restricting the taxonomy to the minimum degree of specificity necessary to support a user experience on a Web site, rather than by attempting to exhaustively describe either the breadth of the field or the distinctions possible within it.

For example, our list of “Medical Complications” is limited to five items—nosocomial infections, pressure ulcers, delirium, venous thrombosis, and falls—rather than an enumeration of every possible medical complication. These five items were selected based on their prominence in the field and the existing literature. While we do sacrifice some specificity in the labeling, we avoid creating the unusable user experience that would result from an attempt to list every possible medical complication.

The taxonomy is organized along 7 descriptive axes: (1) Setting of Care, (2) Target Audience, (3) Clinical Area, (4) Safety Target, (5) Error Types, (6) Approach to Improving Safety, and (7) Resource Origin. Each axis is a hierarchy of terms, ranging from the very general (e.g., Setting of Care > Hospitals) to the very specific (e.g., Safety Target > Medication Safety > Medication Errors > Transcription Errors). Each AHRQ PSNet resource is tagged by professional indexers with zero or more taxonomy terms from each of the seven axes and could be tagged with very specific “leaf” terms, or more general “trunk” terms as appropriate. The tagging drives the site’s “Browse” user experience, where users can conceptually traverse the resource collection along a descriptive axis, or define a targeted search that intersects terms from two or more of the descriptive axes.

In the initial design, the optimum level of specificity was determined subjectively by the editors, with a prejudice toward avoiding an overly complex user experience. As AHRQ PSNet has accumulated content and visitor usage data, we have used statistical methods to identify places in the taxonomy to increase specificity or add new topic areas (e.g., “medication reconciliation” and “red rules” were added after site launch). We anticipate ongoing revisions to the taxonomy, based on statistical analysis, editorial assessment of the trends in the literature, and subjective feedback from the site’s visitors.

Site Usage and User Satisfaction

We have monitored the impact of AHRQ PSNet by three mechanisms:

• Site usage—measured by unique visits per month.
• User content selection—measured by analysis of user search behavior and resources accessed.
• User demographics and satisfaction—measured by voluntary survey responses.

As shown in Figure 3, the site has steadily attracted more users since its launch, with an increase from approximately 30,000 visits per month in April 2005 to more than 110,000 in July 2007. Additional data from July 2007 indicate more than 3,600 site visits daily, more than 6,500 subscribers receiving the weekly “What’s New” e-mail newsletter, and more than 3,200 subscribers with established “My PSNet” accounts. At the launch of AHRQ PSNet, the number of visits to it and to AHRQ’s WebM&M were roughly equal. By way of comparison, AHRQ PSNet now has approximately 33 percent more daily visits than AHRQ WebM&M; combined, the two sites are trending toward approximately 2.5 million unique visits per year.

Figure 3

Visits per month to AHRQ PSNet: April 2005 to July 2007.

Each time a user accesses the site and views a specific resource, Silverchair’s Web server logs a “hit” for the individual resource item, as well as for the taxonomy terms used to classify that item. For example, if a user viewed an annotation of a journal article on the subject of “medication reconciliation,” it would be recorded as a “hit” for the taxon “Medication Safety.” When a user searches the site, the server also logs the exact search string used. We use monthly summary reports of these data to analyze our user’s content interests. Table 1 shows the most commonly accessed taxons over the past year; they represent the patient safety topics of greatest interest to our users. The diversity of topics, ranging from specific errors and interventions (e.g., “medication errors” and “teamwork training”) to systems and organizational issues (e.g., “nurse staffing ratios” and “culture of safety”) speaks to the varied interests of our readers. Last summer, we conducted a voluntary user survey, gathering data on user satisfaction with the site and user demographic data. Most respondents identified their primary institutional affiliation as a hospital or health care system (60 percent) or academic institution (16 percent). The respondents came from a variety of professions and institutional roles, including nurses, physicians, and quality improvement (QI) and patient safety professionals, indicating that AHRQ PSNet is meeting our goal of providing resources for a diverse array of users (Table 2). Overall, users were very satisfied with the content, features, and ease of use of AHRQ PSNet; 92 percent of respondents stated they would recommend the site to a colleague.

Table 1

Most common search strings and taxonomy areas: July 2006 – July 2007.

Table 2

Characteristics of AHRQ PSNet users.