NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau J, Hadar N, Iovin R, et al. Proposed Governance and Data Management Policy for the Systematic Review Data Repository [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Sep.

Cover of Proposed Governance and Data Management Policy for the Systematic Review Data Repository

Proposed Governance and Data Management Policy for the Systematic Review Data Repository [Internet].

Show details

SRDR Governance and Data Management

The SRDR aims to help stakeholders make well-informed health care decisions by maintaining and promoting the accessibility of systematic review data on the effects and relative effectiveness of health care interventions. In order to gain a thorough understanding of the issues underpinning the creation of the proposed repository, we convened a panel of technical advisors with expertise in trial registries, scientific databases, systematic review methodologies, and informatics. Based on this exchange, a number of basic operational principles for the SRDR were articulated as essential to the system:

  1. To maintain an open repository by:
    1. Accepting contributions that meet established scientific standards, made both voluntarily by the research community at large or as required by funders of systematic reviews.
    2. Relying upon community/user review of data (e.g., public posting of proposed corrections or comments).
    3. Making deposited data freely available.
    4. Operating under policies shaped by stakeholder input.
    5. Maintaining an ongoing dialog with systematic review stakeholders to encourage a collaborative and responsive development process.
  2. To minimize the burden on researchers in contributing data to the SRDR by:
    1. Providing for flexible data entry (e.g., offering dynamic, expandable records and extraction forms).
    2. Providing tools for researchers conducting SRs to seamlessly incorporate SRDR data contribution into their normal workflow.

In the course of this conversation, it was speculated that intellectual property (IP) concerns might pose a stumbling block with respect to both these principles. However, we determined that, due to the nature of the SRDR as a repository of pure scientific data, the reuse of information obtained from the SRDR for the purpose of scientific research falls squarely under the fair use exception to copyright law. Regardless, EPCs and other publicly funded institutions and contractors already cede (or license) IP rights on their work to the government. AHRQ or other sponsoring agencies could simply mandate the deposition of extracted data as a stipulation of funding contracts. Participating foreign or private organizations or individuals could be notified of relevant U.S. IP law via assent to SRDR terms of use or even an explicit license, such as one available under the Creative Commons, which is one of several copyright licenses that allow the distribution of copyrighted works. The licenses differ by several combinations that condition the terms of distribution. For more information regarding the Creative Commons license, visit

Aside from standard attribution practices, we believe that IP concerns with regards to the SRDR are minimal, and we do not anticipate any dilemmas with respect to copyright, the deposition of previously published data, or the participation of organizations or individuals with stakes in the future publication of their scientific findings. Consultation with AHRQ legal counsel, however, will continue throughout the development process to proactively address any potential complexities or concerns.

In further discussion with the advisory panel, we determined that, primarily, the key to realizing the above enumerated principles and ensuring the long-term success of the repository, are the following three operational components:

  1. A governing or advisory body
  2. An efficient and discerning user/contributor certification process
  3. An effective means of ensuring quality control

Governing Body

The SRDR governing body would be charged with establishing and managing policies and processes, including those related to data quality control and user certification; and setting overall strategic goals and priorities, and ensuring they are met. We believe that a joint committee of various stakeholders would be best suited in guiding development of the repository, by guaranteeing representation of the needs of the SRDR’s various constituencies. The charter including the specifics of the functioning, role and responsibilities of the governing body would be discussed and decided by the SRDR advisory committee. Drafting an organization charter for this body, which would specify the roles and responsibilities of the governing entity, would be among the tasks of the SRDR advisory committee.

AHRQ has committed to the development of the SRDR for 2 years; however, the funding source beyond this initial period remains to be determined. We expect that the structure of the proposed governing body will differ somewhat based on the eventual funding source (as opposed to the user certification and quality assurance processes). Below, we describe a possible configuration for this body for each of two funding scenarios: (1) The SRDR will continue to be fully funded and managed by the U.S. government; or (2) The SRDR will be funded and operated as an independent nongovernmental entity, either as a standalone organization or as a unit within an academic institution.

We excluded a third scenario, commercial or fee-for-service operation, because we felt it counter to the SRDR’s stated purpose, which is to provide an open and freely accessible repository to promote systematic review research, and hence unlikely to garner sufficient support from stakeholder groups.

Funding Scenario 1–Federal Government

Under this scenario, the daily operation of the system could be assigned to an in-agency team of government personnel or to an outside organization through a number of available mechanisms (e.g., contract, cooperative agreement, grant), such as an EPC or other qualified organization.

Any advisory body created to assist the supervising government officials directly would be subject to the Federal Advisory Committee Act and its procedural requirements, such as publication of notices of meetings in the Federal Register and the payment of members as special government employees. However, the advisory body might be appointed by the chosen contractor to advise the contractor directly, in which case these requirements would not apply. In either circumstance, the advisory body’s primary roles would be unchanged.

The committee would be convened by the sponsoring government agency or contracted organization. Representation is anticipated from: EPCs, Cochrane Collaboration, Center for Reviews and Dissemination (CRD), National Library of Medicine (NLM), technical experts from the informatics community, and other relevant research organizations and user communities. The responsible agency official would select and invite the advisory committee members following consultation with stakeholders, or oversee the contracting organization’s selection process. The agency (or contractor) would also determine appropriate terms of service.

Based on internal institutional experience, we expect that an ideally sized committee would consist of six to nine members (ideal because it is not so large as to make the process unwieldy, but not so small that it precludes an adequate representation of the variety of stakeholder interests), with each member representing a relevant field or constituency (e.g., systematic reviewers, researchers, clinicians, journal editors, patients, biomedical informaticians, policymakers, health insurers). Every so often, the sponsoring agency would post an invitation for volunteers of appropriate backgrounds or representatives of contributor and user communities to serve on the advisory committee. The request could be posted on the Web and other resources (e.g., scientific journals) that serve relevant communities. Each non-Federal agency member would commit to serve for some predetermined time period and participate in a regular conference call. In order to preserve continuity of policy and promote the transfer of experience, we recommend that committee turnover be gradual, such that newly appointed members’ terms would overlap with those of some previous appointees. Representatives of the sponsoring Federal agency would be assigned/reassigned by the director of the respective agency.

In a slight variation on this scenario, the SRDR could be funded by the new legislatively mandated, quasi-public, not-for-profit Patient-Centered Outcomes Research Institute (PCORI). As PCORI is still evolving, it is difficult to anticipate with any certainty the relationships and contractual agreements that could be set in place with regards to the SRDR. PCORI might fund the SRDR directly (through contract for “management of funds”) or through another governmental agency (e.g., AHRQ, NIH). In either situation, the operational governance arrangements would be similar to those outlined above (presumably with an added layer of PCORI oversight). The advisory committee might be formed and selected by PCORI or by the SRDR operational entity (possibly in consultation with PCORI); however, it would not be subject to the Federal Advisory Committee Act, because it would not be exclusively advising Federal officials.

Funding Scenario 2–Independent Nonprofit

In the second scenario, SRDR would operate as an independent nongovernmental entity, either residing within an academic institution or existing as a standalone entity. Financial support would come from nonprofit sources, including public and private grants, donations, and/or other funding mechanisms.

The Cochrane Collaboration might serve as an organizational model (see Cochrane Policy Manual, for such a standalone entity. As in the Cochrane Collaboration, the SRDR’s mission would be accomplished by groups of volunteers. Under this model, members of relevant stakeholder groups—Federal public health and health research agencies, researchers, health care professionals, consumers, and others—will have come together because they share an interest in reliable, up-to-date evidence in health care.

Similar to the Cochrane’s governance model, members of various interest groups would be eligible to participate in selecting members of the governance body (in the form of a board of trustees or a steering committee). This body would meet face-to-face regularly, with working subgroups holding discussions by teleconference between meetings. Governance body decisions would be guided by goals and objectives set out in the SRDR strategic plan. In addition, a methods group could be established to provide scientific support and operations guidance, and a separate group could be responsible for internal administrative functions.

This model could be adapted to a scenario in which much of the responsibility for SRDR is assumed by a single group residing within an academic institution. In such a case, the group would maintain the repository and seek funding through alternate mechanisms. Under these circumstances, the organization might choose to fully adopt the Cochrane model or eschew a governing committee altogether and relegate final operational and policy decisions to itself.

User Certification

Data security and integrity are central concerns for any electronic repository of scientific information. Above all else, users must trust that the data deposited in the SRDR are both secure and accurate; else the repository will be of little use and most certainly fail to grow, let alone thrive.

In order to address these concerns, we propose granting tiered access to SRDR data and functionality, with certification required for higher levels of user privilege. Based on an assessment of probable use cases, our proposed system contains four types of SRDR users, each with a specific set of data access permissions. The proposed user types in order of increasing privilege are: (1) Viewer, (2) Commentator, (3) Contributor, and (4) Publisher/Editor.

  • Viewer: Any consumer of SRDR data. Such users would only be allowed to view and download published data; no commenting, editing, or supplementation of data would be permitted. Registration prior to accessing the SRDR system is voluntary.
  • Commentator: An SRDR user with permission to post comments on published SRDR entries. Such users would have to register on the SRDR Web site, providing basic identifying and contact information, including their name, their affiliated organization, and a valid email address. Following email verification, newly registered commentators would be asked to accept the terms of use for the repository’s discussion features. All comments would be vetted for clarity and adherence to system rules by the SRDR support team prior to posting.
  • Contributor: A registered individual who has been given permission to contribute data to a project housed in the repository. Accounts of this type would initially be assigned via certified groups or organizations, rather than through the SRDR’s Web registration. Initially, only EPCs and other invited groups would be granted rights to contribute data to the SRDR. Individual contributors who wished to submit evidence reviews not undertaken under the auspices of an SRDR-recognized or -affiliated sponsoring organization would be granted separately under policies to be established under guidance from the SRDR governing body.
  • Publisher/Editor: Reserved for one or more individuals designated by the EPC director or equivalent official of other participating organizations, these individuals are granted the authority to “publish” a completed systematic review project on the SRDR Web site (that is, make it publically viewable). Currently, projects are viewable only by their creators and chosen collaborators until designated otherwise. It would be left to the participating organizations to decide who and how many members of the organization would be designated for this role.

In addition to the above user types, we recommend that contributing organizations appoint an SRDR coordinator to serve as his/her organization’s internal SRDR point person. This coordinator would be responsible for organizing and facilitating subsequent training and support for new users within his/her organization, and serving as a liaison to the SRDR support team.

Quality Control

In addition to data security, establishing reliable, comprehensive, and effective quality control processes is crucial to inspire user trust and will encourage SRDR’s long-term success and sustainability. To be clear, with regards to the SRDR, our use of “data” here refers to those extracted from primary scientific publications (or the interpretations/calculations made of/thereupon) and transcribed by systematic reviewers into the SRDR. Below we outline a proposal for a multi-level strategy for quality assurance.

Firstly, during data entry, we propose using:

  1. Per-field error checking (e.g., built in range check for predefined numerical data types)
  2. Process and data visualization (e.g., progress meters, summary table creation and export)

These tools would assist users in entering data accurately during the data extraction process. Some error checking may also be user specified (range and direction for numeric data fields, for example). Process and data visualization would allow project leads to review summary data in order to identify errors or missing data.

For “published” data (those contained within completed SRDR projects that have been made publicly viewable), those entered de novo as well as those imported from projects completed using other evidence review systems (e.g., DistillerSR), we suggest implementing two approaches to facilitate quality assurance:

  1. Regular data audits
  2. Platform for community collaboration including for flagging or commenting on deposited data

We anticipate that the SRDR will be archiving data from a large number of systematic reviews, and that reviewing and curating all deposited records will quickly become infeasible. Regular audits of randomly selected records would strike a reasonable balance between maintaining data quality and ensuring the quality assurance workload remains tractable.

These audits could be conducted by the Tufts EPC initially (as the group most familiar with the repository) and then, going forward, by each certified organization’s SRDR coordinator. Under our proposal, the Tufts EPC would train these coordinators to serve as a liaison between the SRDR support team and his/her organization. During the initial period of his/her tenure, the coordinating editor would be mentored by the SRDR support team, eventually becoming the contributing organization’s local expert. SRDR coordinator responsibilities would include:

  • Providing basic support locally
  • Forwarding more challenging support issues to the SRDR support team
  • Training and mentoring new users within their home organization
  • Curating data deposited by new users under their tutelage

Platform for community collaboration such as Wikis have been identified as potentially useful in a number of scenarios applicable to the systematic review activities sponsored by AHRQ.2 By enabling collaborative commenting and data review, the entire user base can be enlisted in the quality assurance process.

In order to promote a high signal-to-noise ratio in public feedback, we suggest that only registered users be permitted to comment on published records using the SRDR’s discussion tools (the commentator user level and its attendant requirements described above). Users could also be given the opportunity to undergo a short online training session to learn how to use the proposed discussion features prior to submitting comments.

We also recommend that comments made by users be made viewable only after they are reviewed by the support team. Messages that do not adhere to commenting guidelines would not be posted. Alternatively, it may be more expedient to allow live posting and remove offending posts only as needed.

It is essential for the success of the repository that the deposited data are as accurate as possible. We believe that a multilayered, pre- and postdeposition approach would best ensure the SRDR accumulates quality data. In the instances that data are found to be incorrect or not meet minimum reporting standards, and the original contributors of the data cannot be located (or are unable or unwilling to make the corrections), we suggest that the support team reserve the right to retract these records under guidelines developed in partnership with the governing body.


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (176K)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...