NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Gliklich RE, Leavy MB, Dreyer NA, editors. Registries for Evaluating Patient Outcomes: A User’s Guide [Internet]. 4th edition. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020 Sep.

Cover of Registries for Evaluating Patient Outcomes: A User’s Guide

Registries for Evaluating Patient Outcomes: A User’s Guide [Internet]. 4th edition.

Show details

Chapter 1Patient Registries

1. Introduction

The purpose of this document is to serve as a guide for the design and use of patient registries for scientific, clinical, and health policy purposes. Properly designed and executed, patient registries can provide a real-world view of clinical practice, patient outcomes, safety, and comparative effectiveness. This User’s Guide primarily focuses on practical design and operational issues, evaluation principles, and best practices. Where topics are well covered in other materials, references and/or links are provided. The goal of this document is to provide stakeholders in both the public and private sectors with information they can use to guide the design and implementation of patient registries, the analysis and interpretation of data from patient registries, and the evaluation of the quality of a registry or one of its components. Where useful, case examples have been incorporated to illustrate particular points or challenges.

The term registry1 is defined both as the act of recording or registering and as the record or entry itself. Therefore, “registries” can refer to both programs that collect and store data and the records that are so created.

The term patient registry is generally used to distinguish registries focused on health information from other record sets, but there is no consistent definition in current use. E.M. Brooke, in a 1974 publication of the World Health Organization, further delineated registries in health information systems as “a file of documents containing uniform information about individual persons, collected in a systematic and comprehensive way, in order to serve a predetermined purpose.”2

The National Committee on Vital and Health Statistics3 describes registries used for a broad range of purposes in public health and medicine as “an organized system for the collection, storage, retrieval, analysis, and dissemination of information on individual persons who have either a particular disease, a condition (e.g., a risk factor) that predisposes [them] to the occurrence of a health-related event, or prior exposure to substances (or circumstances) known or suspected to cause adverse health effects.”

Other terms also used to refer to patient registries include clinical registries, clinical data registries, disease registries, and outcomes registries.4,5

This User’s Guide focuses on patient registries that are used for evaluating patient outcomes, defined as follows:

  • A patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves one or more stated scientific, clinical, or policy purposes.
  • The patient registry database describes a file (or files) derived from the registry.

It is not intended to address several other types of or uses for registries (although many of the principles may be applicable), such as geographically based population registries (not based on a disease, condition, or exposure); registries created for public health reporting without tracking outcomes (e.g., vaccine registries); or listing registries that are used solely to identify patients with particular diseases in clinical practices but are not used for evaluating outcomes.

Based on these definitions, the User’s Guide focuses on patient registries in which the following are true (although exceptions may apply):

  • The data are collected in a naturalistic manner, such that the management of patients is determined by the caregiver and patient together and not by the registry protocol.
  • The registry is designed to fulfill specific purposes, and these purposes are typically defined before collecting or aggregating the data. In other words, the data collection or aggregation is purpose driven rather than the purpose being data driven (meaning limited to or derived from what is already available in an existing dataset). However, registries may add or change purposes over time.
  • The registry captures data elements with specific and consistent data definitions.
  • The data are collected in a uniform manner for every patient or can be transformed into uniform data for every patient. This consideration refers to both the types of data and the frequency of their collection.
  • The data collected include data derived from and reflective of the clinical status of the patient (e.g., history, examination, laboratory test, or patient-reported data). Registries include the types of data that clinicians would use for the diagnosis and management of patients.
  • It is common for at least one element of registry data collection to be active, meaning that some data are collected or derived specifically for the purpose of the registry (usually collected from the patient or clinician) rather than inferred from sources that are collected for another purpose (administrative, billing, pharmacy databases, etc.). Also, registry data collection may be a specific, but not the exclusive reason data are being collected, such as might occur with registries that incorporate data from electronic health records. This definition also does not exclude the incorporation of other data sources. Registries can be enriched by linkage with extant databases (e.g., to determine deaths and other outcomes or to assess pharmacy use or resource utilization), as discussed in Chapter 6.

Data from patient registries are generally used for studies that address the purpose for which the registry was created. In some respects, such as the collection of detailed clinical and longitudinal followup data, studies derived from the patient registries described in this User’s Guide resemble traditional observational cohort studies. Beyond traditional cohort studies, however, some registry-based studies may be more flexible in that the scope and focus of the data collection activity of the registry may be adapted over time to address additional needs. For example, new studies, such as cluster-randomized studies or case-control studies, may be nested within an ongoing registry, and the database derived from the registry may be used to support secondary studies, such as studies that link the registry database with other data sources to explore new questions.

2. Current Uses for Patient Registries

A patient registry can be a powerful tool to observe the course of disease; to understand variations in treatment and outcomes; to examine factors that influence prognosis and quality of life; to describe care patterns, including appropriateness of care and disparities in the delivery of care; to assess effectiveness; to monitor safety and harm; and to measure quality of care. Through functionalities such as feedback of data, registries are also being used to study quality improvement.6

Different stakeholders perceive and may benefit from the value of registries in different ways. For example, for a clinician, registries can collect data about disease presentation and outcomes on large numbers of patients rapidly, thereby producing a real-world picture of disease, current treatment practices, and outcomes. For a physician organization, a registry might provide data that can be used to assess the number of real-world procedures performed using specific new technique or technology, to examine the degree to which clinicians are managing a disease in accordance with evidence-based guidelines, to evaluate the improvement in quality of life of patients following therapeutic management, to focus attention on specific aspects of a particular disease that might otherwise be overlooked, or to provide data for clinicians to compare themselves with their peers.7 For patients and patient advocacy organizations, a registry may increase understanding of the natural history of a disease, contribute to the development of treatment guidelines, or facilitate research on treatment.8,9 From a payer’s perspective, registries can provide detailed information from large numbers of patients on how procedures, devices, or pharmaceuticals are actually used and on their effectiveness in different populations. This information may be useful for determining coverage policies or informing or supporting value-based care programs.10 For a drug or device manufacturer, a registry-based study might demonstrate the performance of a product in the real world, meet a postmarketing commitment or requirement,11 develop hypotheses, or identify patient populations that will be useful for product development, clinical trials design, and patient recruitment. The U.S. Food and Drug Administration (FDA) has noted that “through the creation of registries, a sponsor can evaluate safety signals identified from spontaneous case reports, literature reports, or other sources, and evaluate the factors that affect the risk of adverse outcomes such as dose, timing of exposure, or patient characteristics.”12

Registries may also generate real-world data and real-world evidence to inform regulatory decision making. The 21st Century Cures Act requires the FDA to develop a program “to evaluate the use of real-world evidence (1) to help support the approval of a new indication for a drug approved under section 505(c); and (2) to help to support or satisfy post-approval study requirements.”13 The FDA defines real-world data (RWD) and real-world evidence (RWE) as “data relating to patient health status and/or the delivery of healthcare routinely collected from a variety of sources. Real-World Evidence (RWE) is the clinical evidence about the usage and potential benefits or risks, of a medical product derived from analysis of RWD. Examples of RWD include data derived from electronic health records (EHRs); medical claims and billing data; data from product and disease registries; patient-generated data, including from in-home-use settings; and data gathered from other sources that can inform on health status, such as mobile devices.14 Some registries have already been used as a source of RWD to support regulatory decision making.1518

The use of patient registries varies by condition, with cancer and cardiovascular disease having a large number of registries and conditions such as developmental delay or dementia, far fewer. Overall, the use of patient registries appears to be active and growing. For example, a review of ClinicalTrials.gov focusing on cancer reveals over 650 entries for patient registries with a wide range of purposes. Of these, four have more than 100,000 participants, and 56 have more than 10,000. In some cases, the drivers for these registries have been Federal stakeholders. Since 2005, the FDA Center for Devices and Radiological Health has called for some 265 post-approval studies, many of which use new or existing registries to study the real-world effectiveness of specific devices in community practice.19

2.1. Evaluating Patient Outcomes

Studies from patient registries and randomized controlled trials (RCTs) have important and complementary roles in evaluating patient outcomes.20 Ideally, patient registries collect data in a comprehensive manner (with few excluded patients) and therefore produce outcome results that may be generalizable to a wide range of patients. They also evaluate care as it is actually provided, because care is not assigned, determined, or even recommended by a protocol. As a result, the outcomes reported may be more representative of what is achieved in real-world practice. Patient registries also offer the ability to evaluate patient outcomes when clinical trials are not practical (e.g., very rare diseases), and they may be the only option when clinical trials are not ethically acceptable. They are a powerful tool when RCTs are difficult to conduct, such as in surgery or when very long-term outcomes are desired.

RCTs are controlled experiments designed to test hypotheses that can ultimately be applied to real-world care. Because RCTs are often conducted under strict constraints, with detailed inclusion and exclusion criteria (and the need for subjects who are willing to be randomized), they are sometimes limited in their generalizability. If RCTs are not generalizable to the populations to which the information will be applied, they may not be sufficiently informative for decision making. Conversely, patient registries that observe real-world clinical practice may collect the information needed to assess patient outcomes in a generalizable way, but interpreting this information correctly requires analytic methodology geared to address the potential sources of bias that challenge observational studies (see Chapter 13). Interpreting patient registry data also requires checks of internal validity and sometimes the use of external data sources to validate key assumptions (such as comparing the key characteristics of registry participants with external sources in order to demonstrate the comparability of registry participants with the ultimate reference population). Patient registries, RCTs, other study designs, and other data sources should all be considered tools in the toolbox for evidence development, each with its own advantages and limitations.21

2.2. Hierarchies of Evidence

One question that arises in a discussion of this type is where to place studies derived from patient registries within the hierarchies of evidence that are frequently used in developing guidelines or decision making. While the definition of patient registry used in this User’s Guide is intentionally broad, the parameters of quality described in Chapter 14 are intended to help the user evaluate and identify registries that are sufficiently rigorous observational studies for use as evidence in decision making. Many registries are, or include, high-quality studies of cohorts designed to address a specific problem and hypothesis. Still, even the most rigorously conducted registries, like prospective observational studies, are traditionally placed in a subordinate position to RCTs in some commonly used hierarchies, although equal to RCTs in others.2224 Debate continues in the evidence community regarding these traditional methods of grading levels of evidence, their underlying assumptions, their shortcomings in assessing certain types of evidence (e.g., benefit vs. harm), and their interscale consistency in evaluating the same evidence.21,25,26

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group has proposed a more robust approach that addresses some of the decision-making issues described in this User’s Guide. As noted by the GRADE collaborators:

[R]andomised trials are not always feasible and, in some instances, observational studies may provide better evidence, as is generally the case for rare adverse effects. Moreover, the results of randomised trials may not always be applicable—for example, if the participants are highly selected and motivated relative to the population of interest. It is therefore essential to consider study quality, the consistency of results across studies, and the directness of the evidence, as well as the appropriateness of the study design.27

AHRQ has also developed a guidance system for grading the strength of evidence that recommends a careful assessment of the potential value of observational studies. The guidance, which is designed to support the systematic reviews conducted by the Evidence-based Practice Center (EPC) program, is conceptually similar to the GRADE system.28 When using the AHRQ approach, reviewers typically give evidence from observational studies a low starting grade and evidence from RCTs a high starting grade. These initial grades can then be raised or lowered depending on the strength of the five required evidence domains (study limitations, directness, consistency, precision, and reporting bias).29 For example, the reviewers may find that observational studies are particularly relevant for some systematic review questions. The report notes:

EPCs may act on the judgment that, for certain outcomes such as harms, observational studies have less risk of bias than do RCTs or that the available RCTs have a substantial risk of bias. In such instances, the EPC may move up the initial grade for strength of evidence based on observational studies to moderate or move down the initial rating based on RCTs to moderate.28

Reviewers may also raise or lower evidence grades based on a secondary set of domains (dose-response association, existence of confounding that would diminish an observed effect, and strength of association). These secondary domains supplement the required domains and are used when relevant to the systematic review question. The report explains that the secondary domains “may increase strength of evidence and are especially relevant for observational studies where one may begin with a lower overall strength of evidence grade based on study limitations.”28

As the methods for grading evidence for different purposes continue to evolve, this User’s Guide can serve as a guide to help such evaluators understand study quality and identify well-designed registries. Beyond the evidence hierarchy debate, users of evidence understand the value of registries for providing complementary information that can extend the results of clinical trials to populations not studied in those trials, for demonstrating the real-world effects of treatments outside of the research setting (potentially in large subsets of affected patients), and for providing long-term followup when such data are not available from clinical trials.

2.3. Defining Patient Outcomes

The focus of this User’s Guide is the use of registries to evaluate patient outcomes. An outcome may be thought of as an end result of a particular healthcare practice or intervention. According to AHRQ, end results include effects that people experience and care about.30 The National Cancer Institute further clarifies that “final” endpoints are those that matter to decision-makers: patients, providers, private payers, government agencies, accrediting organizations, or society.31 Examples of these outcomes include biomedical outcomes, such as survival and disease-free survival, health-related quality of life, satisfaction with care, and economic burden.32 Although final endpoints are ultimately what matter, it is sometimes more practical when creating registries to collect intermediate outcomes (such as whether processes or guidelines were followed) and clinical outcomes (such as whether a tumor regressed or recurred) that predict success in improving final endpoints. Clinical outcomes are particularly important in chronic conditions such as asthma and diabetes, where the treatment intent is management.

In Crossing the Quality Chasm,33 the Institute of Medicine (now the National Academy of Medicine) describes the six guiding aims of healthcare as providing care that is safe, effective, efficient, patient-centered, timely, and equitable. (The last three aims focus on the delivery and quality of care.) While these aims are not outcomes per se, they generally describe the dimensions of results that matter to decision-makers in the use of a healthcare product or service: Is it safe? Does it produce greater benefit than harm? Is it clinically effective? Does it produce the desired effect in real-world practice? Does the right patient receive the right therapy or service at the right time? Is it cost effective or efficient? Does it produce the desired effect at a reasonable cost relative to other potential expenditures? Is it patient oriented, timely, and equitable? Most of the patient outcomes that registries evaluate reflect one or more of the six guiding aims. For example, a patient presenting with an ischemic stroke to an emergency room has a finite window of opportunity to receive a thrombolytic drug, and the patient outcome, whether the patient achieves full recovery, is dependent not only on the product’s dissolving the clot but also on the timeliness of its delivery.34,35

When defining patient outcomes, as well as other data elements, registries are advised to consider using existing core or minimum sets of outcome measures and common data elements whenever possible. Core or minimum sets of outcome measures specify and define the critical outcomes of interest in a disease or condition area; these recommendations generally are developed through consensus-based processes. Common data elements (CDEs) are standardized terms that can be used in multiple research settings. Core sets of outcome measures and CDEs can be found in the scientific literature. Some initiatives, such as the National Library of Medicine’s CDE Resource Portal36 and the COMET Initiative,37 compile information on existing CDEs and core outcome sets. Use of core sets of outcome measures and CDEs is intended to facilitate comparing, aggregating, and linking data across registries and other research efforts. AHRQ has funded the development of some minimum sets of outcome measures specific to registries through the Outcome Measures Framework (OMF) project.38,39 The OMF is a common model for measuring patient and clinician relevant outcomes across different conditions for registries and clinical practice.23 Through a series of collaborative stakeholder and patient registry working groups, minimum measure sets are or have been developed and published for several conditions.

2.4. Purposes of Registries

As discussed throughout this User’s Guide, registries should be designed and evaluated with respect to their intended purpose(s). Registry purposes can be broadly described in terms of patient outcomes. While there are a number of potential purposes for registries, this handbook primarily discusses four major purposes: (1) describing the natural history of disease, (2) determining clinical and/or cost-effectiveness, (3) assessing safety or harm, and (4) measuring or improving quality of care. Other purposes of patient registries mentioned but not discussed in detail in this User’s Guide are public health surveillance and disease control. An extensive body of literature from the last half century of experience with cancer and other disease surveillance registries is available.

2.4.1. Describing Natural History of Disease

Registries may be established to evaluate the natural history of a disease, meaning its characteristics, management, and outcomes with and/or without treatment. The natural history may be variable across different groups or geographic regions, and it often changes over time. In many cases, the natural histories of diseases are not well described. Furthermore, the natural histories of diseases may change after the introduction of certain therapies. As an example, patients with rare diseases, such as the lysosomal storage diseases, who did not previously survive to their 20s, may now be entering their fourth and fifth decades of life, and this uncharted natural history is being first described through a registry.40 An ancillary purpose of some registries tracking the natural history of a disease is to facilitate identification of participants for potential recruitment into clinical trials.41

2.4.2. Determining Effectiveness

Registries may be developed to determine clinical effectiveness or cost-effectiveness in real-world clinical practice. Multiple studies have demonstrated disparities between the results of clinical trials and results in actual clinical practice.42,43 Furthermore, efficacy in a clinical trial for a well-defined population may not be generalizable to other populations or subgroups of interest. As an example, many important heart failure trials have focused on a predominantly white male population with a mean age of approximately 60 years, whereas actual heart failure patients are older, more diverse, and have a higher mortality rate than the patients in these trials.44 Similarly, underrepresentation of older patients has been reported in clinical trials of 15 different types of cancer (e.g., studies with only 25 percent of patients age 65 years and over, while the expected rate is greater than 60 percent).45 Data from registries have been used to fill these gaps for decision-makers. For example, the FDA used the American Academy of Ophthalmology’s intraocular lens registry to expand the label for intraocular lenses to younger patients.46 Registries may also be particularly useful for tracking effectiveness outcomes for a longer period than is typically feasible with clinical trials. For example, some growth hormone registries have tracked children well into adulthood.

In addition to clinical effectiveness, registries can be used to assess cost-effectiveness. Registries can be designed to collect cost data and effectiveness data for use in modeling cost-effectiveness.47 Cost-effectiveness is a means to describe the comparative value of a healthcare product or service in terms of its ability to achieve a desired outcome for a given unit of resources.48 A cost-effectiveness analysis examines the incremental benefit of a particular intervention and the costs associated with achieving that benefit. Cost-effectiveness studies compare costs with clinical outcomes measured in units such as life expectancy or disease-free periods. Cost-utility studies compare costs with outcomes adjusted for quality of life (utility), such as quality-adjusted life years (QALYs). Utilities allow comparisons to be made across conditions because the measurement is not disease specific.49 It should be noted that for both clinical effectiveness and cost-effectiveness, differences between treatments are indirect and must be inferred from data analysis, simulation modeling, or some mixture.

With improvement in methodologies, including better methods for managing bias and better understanding of the limitations, there is increasing interest and investment from many stakeholders in registries for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). The Patient-Centered Outcomes Research Institute (PCORI) has defined PCOR as research that “assesses the benefits and harms of preventive, diagnostic, therapeutic, palliative, or health delivery system interventions to inform decision making, highlighting comparisons and outcomes that matter to people; is inclusive of an individual’s preferences, autonomy and needs, focusing on outcomes that people notice and care about such as survival, function, symptoms, and health-related quality of life; incorporates a wide variety of settings and diversity of participants to address individual differences and barriers to implementation and dissemination; and investigates (or may investigate) optimizing outcomes while addressing burden to individuals, availability of services, technology, and personnel, and other stakeholder perspectives.”50 PCORI has recognized registries as an important potential source of data to support PCOR, in large part because of their ability to provide information on ‘real-world’ settings and broad patient populations. PCORI included minimum standards for the use of registries for PCOR in the Methodology Report.51 While some registries are designed explicitly to examine questions of comparative effectiveness or patient-centered outcomes research, many others are designed for different objectives yet still collect data that are useful for these analyses. Registries that were not explicitly designed for CER or PCOR may need to be augmented or linked to other data sources—for example, to obtain long-term outcomes data in the case of an in-hospital registry using linkage to claims data to evaluate blood pressure medications.52

2.4.3. Measuring or Monitoring Safety and Harm

Registries may be created to assess safety versus harm. Safety here refers to the concept of being free from danger or hazard. One goal of registries in this context may be to quantify risk or to attribute it properly. Broadly speaking, patient registries can serve as an active surveillance system for the occurrence of unexpected or harmful events for products and services. Such events may range from patient complaints about minor side effects to severe adverse events such as fatal drug reactions or patient falls in the hospital.

Patient registries offer multiple advantages for active surveillance. First, the current practice of spontaneous reporting of adverse events relies on a nonsystematic recognition of an adverse event by a clinician and the clinician’s active effort to make a report to manufacturers and health authorities. (In the United States, patients may also report adverse events directly to the FDA.) Second, these events are generally reported without a denominator (i.e., the exposed or treated population), and therefore an incidence rate is difficult to determine. Because patient registries can provide systematic data on adverse events and the incidence of these events, they are being used with increasing frequency in the areas of healthcare products and services. For example, the FDA recently established the National Evaluation System for health Technology (NEST) to integrate data from patient registries, electronic health records, and administrative claims data to efficiently generate evidence of medical device safety and effectiveness.53

2.4.4. Measuring Quality

Registries may be created to measure quality of care. The Institute of Medicine defined quality as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.” Quality-focused registries are being used increasingly to assess differences between providers or patient populations based on performance measures that compare treatments provided or outcomes achieved with “gold standards” (e.g., evidence-based guidelines) or comparative benchmarks for specific health outcomes (e.g., risk-adjusted survival or infection rates). Such programs may be used to identify disparities in access to care, demonstrate opportunities for improvement, establish differentials for payment by third parties, or provide transparency through public reporting. There are multiple examples of such differences in treatment and outcomes of patients in a range of disease areas.5458 As healthcare transitions from quality measurement focused on adherence with evidence-based practice standards to longitudinal, standardized, patient-level outcomes assessment, registries will play an important role.

2.4.5. Multiple Purposes

Many registries will be developed to serve more than one of these purposes. Registries developed for one purpose may also be modified to serve additional purposes as the research, practice, or policy environment changes. While registries often serve more than one purpose, their original or primary purpose generally guides their design and, as a result, more care is needed in evaluating results for secondary or additional purposes.

3. Taxonomy for Patient Registries

Even limited to the definitions described above, the breadth of studies that might be described as patient registries is large. Patients in a registry are typically selected based on a particular disease, condition (e.g., a risk factor), or exposure. This User’s Guide uses these common selection criteria to develop a taxonomy or classification based on how the populations for registries are defined. Three general categories with multiple subcategories and combinations account for the majority of registries that are developed for evaluating patient outcomes. These categories include observational studies in which the patient has had an exposure to a product or service, or has a particular disease or condition, or various combinations thereof.

3.1. Product Registries

In the case of a product registry, the patient is exposed to a healthcare product, such as a drug or a device, or other potential risk factor (e.g., environmental or personal exposure). The exposure may be brief, as in a single dose of a pharmaceutical product, or extended, as in an implanted device or chronic usage of a medication.

Device registries may include all, or a subset, of patients who receive the device. A registry for all patients who receive an implantable cardioverter defibrillator, a registry of patients with hip prostheses, or a registry of patients who wear contact lenses are all examples of device registries. Biopharmaceutical product registries similarly have several archetypes, which may include all, or subsets, of patients who receive the biopharmaceutical product. For example, the British Society for Rheumatology established a national registry of patients on biologic therapy.59 Again, the duration of exposure may range from a single event to a lifetime of use. Eligibility for the registry includes the requirement that the patient received the product or class of products (e.g., COX-2 inhibitors). In some cases, public health authorities mandate such registries to ensure safe use of medications. Examples include registries for thalidomide, clozapine, and isotretinoin.

Pregnancy registries represent a separate class of biopharmaceutical product registries that focus on possible exposures during pregnancy and the neonatal consequences. The FDA has a specific guidance focused on pregnancy exposure registries.60

3.2. Health Services Registries

In the context of evaluating patient outcomes, another type of exposure that can be used to define registries is exposure to a healthcare service. Healthcare services that may be used to define inclusion in a registry include individual clinical encounters, such as office visits or hospitalizations, procedures, or full episodes of care. Examples include registries enrolling patients undergoing a procedure (e.g., carotid endarterectomy, appendectomy, or primary coronary intervention) or admitted to a hospital for a particular diagnosis (e.g., community-acquired pneumonia). In these registries, one purpose of the registry is to evaluate the healthcare service with respect to the outcomes. Healthcare service registries are sometimes used to evaluate the processes and outcomes of care for quality measurement purposes (e.g., Get With The Guidelines® of the American Heart Association, National Surgical Quality Improvement Program of the Department of Veterans Affairs and the American College of Surgeons).

3.3. Disease or Condition Registries

Disease or condition registries use the state of a particular disease or condition as the inclusion criterion. In disease or condition registries, the patient may always have the disease (e.g., a rare disease such as cystic fibrosis or Pompe disease, or a chronic illness such as heart failure, diabetes, or end-stage renal disease) or may have the disease or condition for a more limited period of time (e.g., infectious diseases, some cancers, obesity). These registries typically enroll the patient at the time of a routine healthcare service, although patients also can be enrolled through voluntary self-identification processes that do not depend on utilization of healthcare services (such as Internet recruiting of volunteers). In other disease registries, the patient has an underlying disease or condition, such as atherosclerotic disease, but is enrolled only at the time of an acute event or exacerbation, such as hospitalization for a myocardial infarction or ischemic stroke.

3.4. Combinations

Complicating this classification approach is the reality that these categories can be overlapping in many registries. For example, a patient with ischemic heart disease may have an acute myocardial infarction and undergo a primary coronary intervention with placement of a drug-eluting stent and postintervention management with clopidogrel. This patient could be enrolled in an ischemic heart disease registry tracking all patients with this disease over time, a myocardial infarction registry that is collecting data on patients who present to hospitals with acute myocardial infarction (cross-sectional data collection), a primary coronary intervention registry that includes management with and without devices, a coronary artery stent registry limited to ischemic heart disease patients, or a clopidogrel product registry that includes patients undergoing primary coronary interventions.

3.5. Duration of Observation

The duration of the observational period for a registry is also a useful descriptor. Observation periods may be limited to a single episode of care (e.g., a hospital discharge registry for diverticulitis), or they may extend for as long as the lifetime of patients with a chronic disease (e.g., cystic fibrosis or Pompe disease) or patients receiving a novel therapy (e.g., gene therapy). The period of observation or followup depends on the outcomes of interest.

3.6. From Registry Purpose to Design

As will be discussed extensively in this document, the purpose of the registry defines the registry focus (e.g., product vs. disease) and therefore the registry type. A registry created for the purpose of evaluating outcomes of patients receiving a particular coronary artery stent might be designed as a single product registry if, for example, the purpose is to systematically collect adverse event information on the first 10,000 patients receiving the product. However, the registry might alternatively be designed as a healthcare service registry for primary coronary intervention if the purpose is to collect comparative effectiveness or safety data on other treatments or products within the same registry.

4. Patient Registries and Policy Purposes

In addition to the growth of patient registries for scientific and clinical purposes, registries are receiving increased attention for their potential role in policymaking or decision making. As stated earlier, registries may offer a view of real-world healthcare that is typically inaccessible from clinical trials or other data sources and may provide information on the generalizability of the data from clinical trials to populations not studied in those trials.

The utility of registry data for decision making is related to three factors: the stakeholders, the primary scientific question, and the context. The stakeholders are those associated with the disease or procedure that may be affected from a patient, provider, payer, regulator, or other perspective. The primary scientific question for a registry may relate to effectiveness, safety, or practice patterns. The context includes the scientific context (e.g., previous randomized trials and modeling efforts that help to more precisely define the primary scientific question), as well as the political, regulatory, funding, and other issues that provide the practical parameters around which the registry is developed. In identifying the value of information from registries, it is essential to look at the data with specific reference to the purpose and focus of the registry.

From a policy perspective, there are several scenarios in which the decision to develop a registry may arise. One possible scenario is as follows. An item or service is considered for use. Stakeholders in the decision collaboratively define “adequate data in support of the decision at hand.” Here, “adequate data” refers to information of sufficient relevance and quality to permit an informed decision. An evidence development strategy is selected from one of many potential strategies (RCT, practical clinical trial, registry, etc.) based on the quality of the evidence provided by each design, as well as the burden of data collection and the cost that is imposed. This tradeoff of the quality of evidence versus cost of data collection for each possible design is termed the “value of information” exercise (Figure 1-1). Registries should be preferred in those circumstances where they provide sufficiently high-quality information for decision making at a sufficiently low cost (relative to other “acceptable” designs).

One set of policy determinations that may be informed by a patient registry centers on the area of payment for items or services. For example, the Centers for Medicare & Medicaid Services (CMS) issued Guidance on National Coverage Determinations With Data Collection as a Condition of Coverage in 2006. That original guidance document (which has undergone subsequent revisions, most recently in 2014) provided several examples of how data collected in a registry might be used in the context of coverage determinations. As described in the Guidance:

[T]he purpose of CED [Coverage with Evidence Development] is to generate data on the utilization and impact of the item or service evaluated in the NCD [National Coverage Determination], so that Medicare can (a) document the appropriateness of use of that item or service in Medicare beneficiaries under current coverage; (b) consider future changes in coverage for the item or service; (c) generate clinical information that will improve the evidence base on which providers base their recommendations to Medicare beneficiaries regarding the item or service.61

The Guidance provided insight into when registry data may be useful to policymakers. These purposes range from demonstrating that a particular item or service was provided appropriately to patients meeting specific characteristics, to collecting new information that is not available from existing clinical trials. CED based on registries may be especially relevant when current data do not address relevant outcomes for beneficiaries, off-label or unanticipated uses, important patient subgroups, or operator experience or other qualifications. Registry-based studies may also be important when an existing treatment is being reconsidered. (An RCT may not be possible under such circumstances.) Registry-based studies are also being used increasingly in fulfillment of postmarketing commitments and requirements.

In many countries, policy determinations on payment rely on cost-effectiveness and cost-utility data and therefore can be informed by registries as well as clinical trials.62 These data are used and reviewed in a variety of ways. In some countries, there may be a threshold above which a payer is willing to pay for an improvement in patient outcomes.63 In these scenarios—particularly for rare diseases, when it can be difficult to gather clinical effectiveness data together with quality-of-life data in a utility format—the establishment of disease-specific data registries has been recommended to facilitate the process of technology assessment and improving patient care.63 In fact, the use of new or existing registries to assess health technology or risk-sharing arrangements is growing in such countries as the United Kingdom, France, Germany, and Australia, and in conditions ranging from bariatric surgery to stroke care.6467

This figure depicts a value of information exercise using a series of boxes connected with arrows. The boxes, in sequence, read, ‘item/service presented for consideration;’ ‘clinical/policy questions formulated;’ adequate data defined; current evidence reviewed; decision made, if data are deemed adequate for decision-making; OR ‘list of alternative designs constructed’, value of information exercise performed; evidence development strategy implemented.

Figure 1-1

Deciding when to develop a registry: The “value of information” exercise.

Consider the clinical question of carotid endarterectomy surgery for patients with a high degree of stenosis of the carotid artery. Randomized trials, using highly selected patients and surgeons, indicate a benefit of surgery over medical management in the prevention of stroke. However, that benefit may be exquisitely sensitive to the surgical complication rates; a relatively small increase in the rate of surgical complications is enough to make medical management the preferred strategy instead. In addition, the studies of surgical performance in a variety of hospitals may suggest substantial variation in surgical mortality and morbidity for this procedure. In such a case, a registry to evaluate treatment outcomes, adjusted by hospital and surgeon, might be considered to support a policy decision as to when the procedure should be reimbursed (e.g., only when performed in medical centers resembling those in the various randomized trials, or only by surgeons or facilities with an acceptably low rate of complications).68

5. Global Registries

As many stakeholders have international interests in diseases, conditions, and healthcare products and services, it is not surprising that interest in global patient registries is growing. While some of the specific legal and regulatory discussions in this User’s Guide are intended for and limited to the United States, most of the concepts and specifics are more broadly applicable to similar activities worldwide. Chapter 7 (ethics, data ownership, and privacy), Chapter 8 (informed consent), and Chapter 12 (adverse event detection, processing, and reporting) are perhaps the most limited in their applicability outside the United States. There may be additional considerations in data element selection and patient-reported outcome measure selection (Chapter 5) stemming from differences ranging from medical training to use of local remedies; the types of data sources that are available outside the United States (Chapter 6); the issues surrounding clinician and patient recruitment and retention in different health systems and cultures (Chapter 10); and specific data collection and management options and complexities (Chapter 11), ranging from available technologies to languages. It is also important to note that this User’s Guide does not address the new European Union (EU) Clinical Trials Regulation scheduled to go into effect in 2019.

6. Future of Registries

With the growing availability of digitized healthcare data and specialized technologies such as natural language processing and machine learning, and the ability to link different data sources from different settings, registries are rapidly evolving. These technologies have greatly advanced and automated patient registry data collection and analyses and are enabling more timely and comprehensive real-world data for a number of the purposes described above. Coupled with standardized outcomes as described later in this document, this evolution of registries will support more timely feedback of critical information to healthcare stakeholders for decision making and will foster the development of learning healthcare systems.69

7. Summary

A patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure and that serves stated scientific, clinical, or policy purpose(s). Studies derived from well-designed and well-performed patient registries can provide a real-world view of clinical practice, patient outcomes, safety, and clinical, comparative, and cost-effectiveness, and can serve a number of evidence development and decision-making purposes. In the chapters that follow, this User’s Guide presents practical design and operational issues, evaluation principles, and good registry practices.

References for Chapter 1

1.
Jacobs V, Cutler MJ, Day JD, et al. Atrial fibrillation and dementia. Trends Cardiovasc Med. 2015;25(1):44–51. PMID: 25442732. DOI: 10.1016/j.tcm.2014.09.002. [PubMed: 25442732] [CrossRef]
2.
Brooke EM. The current and future use of registers in health information systems. Geneva: World Health Organization; 1974. Report No.: Publication No. 8.
3.
Kirkham JJ, Clarke M, Williamson PR. A methodological approach for assessing the uptake of core outcome sets using ClinicalTrials​.gov: findings from a review of randomised controlled trials of rheumatoid arthritis. BMJ. 2017;357:j2262. PMID: 28515234. DOI: 10.1136/bmj.j2262. [PMC free article: PMC5434459] [PubMed: 28515234] [CrossRef]
4.
Dokholyan RS, Muhlbaier LH, Falletta JM, et al. Regulatory and ethical considerations for linking clinical and administrative databases. Am Heart J. 2009;157(6):971–82. PMID: 19464406. DOI: 10.1016/j.ahj.2009.03.023. [PubMed: 19464406] [CrossRef]
5.
Hammill BG, Hernandez AF, Peterson ED, et al. Linking inpatient clinical registry data to Medicare claims data using indirect identifiers. Am Heart J. 2009;157(6):995–1000. PMID: 19464409. DOI: 10.1016/j.ahj.2009.04.002. [PMC free article: PMC2732025] [PubMed: 19464409] [CrossRef]
6.
LaBresh KA, Gliklich R, Liljestrand J, et al. Using “get with the guidelines” to improve cardiovascular secondary prevention. Jt Comm J Qual Saf. 2003;29(10):539–50. PMID: 14567263. [PubMed: 14567263]
7.
Kennedy L, Craig AM. Global registries for measuring pharmacoeconomic and quality-of-life outcomes: focus on design and data collection, analysis and interpretation. Pharmacoeconomics. 2004;22(9):551–68. PMID: 15209525. DOI: 10.2165/00019053-200422090-00001. [PubMed: 15209525] [CrossRef]
8.
Charrow J, Esplin JA, Gribble TJ, et al. Gaucher disease: recommendations on diagnosis, evaluation, and monitoring. Arch Intern Med. 1998;158(16):1754–60. PMID: 9738604. [PubMed: 9738604]
9.
Bushby K, Lynn S, Straub T, et al. Collaborating to bring new therapies to the patient--the TREAT-NMD model. Acta Myol. 2009;28(1):12–5. PMID: 19772190. [PMC free article: PMC2859629] [PubMed: 19772190]
10.
Dhruva SS, Phurrough SE, Salive ME, et al. CMS’s landmark decision on CT colonography--examining the relevant data. N Engl J Med. 2009;360(26):2699–701. PMID: 19474419. DOI: 10.1056/NEJMp0904408. [PubMed: 19474419] [CrossRef]
11.
Mozaffarian D, Benjamin EJ, Go AS, et al. Heart disease and stroke statistics--2015 update: a report from the American Heart Association. Circulation. 2015;131(4):e29–322. PMID: 25520374. DOI: 10.1161/CIR.0000000000000152. [PubMed: 25520374] [CrossRef]
12.
Sanchez JM, Ballinger B, Olgin JE, et al., editors. AF Detection and Ablation Outcomes: Answering Questions That Matter to Patients: Detecting Atrial Fibrillation using a Smart Watch – the mRhythm study. HRS Heart Rhythm 2017; 2017 May 11; Chicago, IL.
13.
45 CFR 164.514 (a)-(c).
14.
U.S. Food and Drug Administration. Framework for FDA’s Real-World Evidence Program. https://www​.fda.gov/media​/120060/download. Accessed June 3, 2019.
15.
Jarow JP, LaVange L, Woodcock J. Multidimensional Evidence Generation and FDA Regulatory Decision Making: Defining and Using “Real-World” Data. JAMA. 2017;318(8):703–4. PMID: 28715550. DOI: 10.1001/jama.2017.9991. [PubMed: 28715550] [CrossRef]
16.
Shuren J., Zuckerman B. How Creative FDA Regulation Led to First-in-the-World Approval of a Cutting-Edge Heart Valve. FDA Voice. U.S. Food and Drug Administration. June 14, 2017. https://blogs​.fda.gov/fdavoice/index​.php​/tag/transcatheter-aortic-valve-replacement-tavr/. Accessed January 17, 2019.
17.
Lutonix® 035 Drug Coated Balloon PTA Catheter. Summary of Safety and Effectiveness Data. U.S. Food and Drug Administration. February 7, 2017. https://www​.accessdata​.fda.gov/cdrh_docs/pdf13/P130024S009B​.pdf. Accessed June 4, 2019.
18.
Gliklich RE, Leavy MB. Assessing Real-World Data Quality: The Application of Patient Registry Quality Criteria to Real-World Data and Real-World Evidence. Ther Innov Regul Sci. 2019:2168479019837520. PMID: 30917687. DOI: 10.1177/2168479019837520. [PubMed: 32072577] [CrossRef]
19.
20.
Dreyer NA, Garner S. Registries for robust evidence. JAMA. 2009;302(7):790–1. PMID: 19690313. DOI: 10.1001/jama.2009.1092. [PubMed: 19690313] [CrossRef]
21.
Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92. PMID: 10861325. DOI: 10.1056/NEJM200006223422507. [PMC free article: PMC1557642] [PubMed: 10861325] [CrossRef]
22.
Guyatt GH, Sackett DL, Sinclair JC, et al. Users’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA. 1995;274(22):1800–4. PMID: 7500513. [PubMed: 7500513]
23.
Gliklich RE, Leavy MB, Karl J, et al. A framework for creating standardized outcome measures for patient registries. J Comp Eff Res. 2014;3(5):473–80. PMID: 25350799. DOI: 10.2217/cer.14.38. [PubMed: 25350799] [CrossRef]
24.
Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Med. 2008;5(3):e67. PMID: 18336067. DOI: 10.1371/journal.pmed.0050067. [PMC free article: PMC2265762] [PubMed: 18336067] [CrossRef]
25.
Atkins D, Eccles M, Flottorp S, et al. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res. 2004;4(1):38. PMID: 15615589. DOI: 10.1186/1472-6963-4-38. [PMC free article: PMC545647] [PubMed: 15615589] [CrossRef]
26.
Rawlins M. De Testimonio: on the evidence for decisions about the use of therapeutic interventions. Clin Med (Lond). 2008;8(6):579–88. PMID: 19149278. DOI: 10.7861/clinmedicine.8-6-579. [PMC free article: PMC4954394] [PubMed: 19149278] [CrossRef]
27.
Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ. 2004;328(7454):1490. PMID: 15205295. DOI: 10.1136/bmj.328.7454.1490. [PMC free article: PMC428525] [PubMed: 15205295] [CrossRef]
28.
Berkman ND, Lohr KN, Ansari M, et al. Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. [PubMed: 24404627]
29.
Viswanathan M, Ansari MT, Berkman ND, et al. Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008.
30.
Clancy CM, Eisenberg JM. Outcomes research: measuring the end results of health care. Science. 1998;282(5387):245–6. PMID: 9841388. [PubMed: 9841388]
31.
Lipscomb J, Snyder CF. The Outcomes of Cancer Outcomes Research: focusing on the National Cancer Institute’s quality-of-care initiative. Med Care. 2002;40:(6 Suppl):III3–10. PMID: 12064755. [PubMed: 12064755]
32.
Lipscomb J, Donaldson MS, Hiatt RA. Cancer outcomes research and the arenas of application. J Natl Cancer Inst Monogr. 2004(33):1–7. PMID: 15504917. DOI: 10.1093/jncimonographs/lgh038. [PubMed: 15504917] [CrossRef]
33.
Hurtado MP, Swift EK, Corrigan JM. Crossing the quality chasm: a new health system for the 21st Century. Washington DC: National Academy Press, Institute of Medicine; 2001.
34.
Schwamm LH, LaBresh KA, Pan WQ, et al. Get With The Guidelines – Stroke produces sustainable improvements in hospital-based acute stroke care. Stroke. 2006;37(2):624.
35.
Schwamm LH, LaBresh KA, Pan WQ, et al. Get With The Guidelines – Stroke improves the rate of “defect-free” acute stroke care. Stroke. 2006;37(2):659.
36.
NIH Common Data Elements (CDE) Repository. National Library of Medicine. National Institutes of Health. https://cde​.nlm.nih.gov/. Accessed June 4, 2019.
37.
The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. http://www​.comet-initiative.org/. Accessed June 4, 2019.
38.
Gliklich RE, Castro M, Leavy MB, et al. Harmonized outcome measures for use in asthma patient registries and clinical practice. J Allergy Clin Immunol. 2019. PMID: 30857981. DOI: 10.1016/j.jaci.2019.02.025. [PubMed: 30857981] [CrossRef]
39.
Calkins H, Gliklich RE, Leavy MB, et al. Harmonized outcome measures for use in atrial fibrillation patient registries and clinical practice: Endorsed by the Heart Rhythm Society Board of Trustees. Heart Rhythm. 2019;16(1):e3–e16. PMID: 30449519. DOI: 10.1016/j.hrthm.2018.09.021. [PubMed: 30449519] [CrossRef]
40.
Barranger JA, O’Rourke E. Lessons learned from the development of enzyme therapy for Gaucher disease. J Inherit Metab Dis. 2001;24: Suppl 2:89–96; discussion 87–8. PMID: 11758684. [PubMed: 11758684]
41.
Tan MH, Thomas M, MacEachern MP. Using registries to recruit subjects for clinical trials. Contemp Clin Trials. 2015;41:31–8. PMID: 25545027. DOI: 10.1016/j.cct.2014.12.012. [PMC free article: PMC4380621] [PubMed: 25545027] [CrossRef]
42.
Wennberg DE, Lucas FL, Birkmeyer JD, et al. Variation in carotid endarterectomy mortality in the Medicare population: trial hospitals, volume, and patient characteristics. JAMA. 1998;279(16):1278–81. PMID: 9565008. [PubMed: 9565008]
43.
MacIntyre K, Capewell S, Stewart S, et al. Evidence of improving prognosis in heart failure: trends in case fatality in 66 547 patients hospitalized between 1986 and 1995. Circulation. 2000;102(10):1126–31. PMID: 10973841. [PubMed: 10973841]
44.
Konstam MA. Progress in heart failure Management? Lessons from the real world. Circulation. 2000;102(10):1076–8. PMID: 10973832. [PubMed: 10973832]
45.
Hutchins LF, Unger JM, Crowley JJ, et al. Underrepresentation of patients 65 years of age or older in cancer-treatment trials. N Engl J Med. 1999;341(27):2061–7. PMID: 10615079. DOI: 10.1056/NEJM199912303412706. [PubMed: 10615079] [CrossRef]
46.
Opthalmic devices and clinical epidemiology. In: Brown SL, Bright RA, Tavris DR, editors. Medical device epidemiology and surveillance: John Wiley & Sons, Ltd.; 2007.
47.
Lipscomb J, Yabroff KR, Brown ML, et al. Health care costing: data, methods, current applications. Med Care. 2009;47(7 Suppl 1):S1–6. PMID: 19536004. DOI: 10.1097/MLR.0b013e3181a7e401. [PubMed: 19536004] [CrossRef]
48.
Eichler HG, Kong SX, Gerth WC, et al. Use of cost-effectiveness analysis in health-care resource allocation decision-making: how are cost-effectiveness thresholds expected to emerge? Value Health. 2004;7(5):518–28. PMID: 15367247. DOI: 10.1111/j.1524-4733.2004.75003.x. [PubMed: 15367247] [CrossRef]
49.
Palmer AJ. Health economics--what the nephrologist should know. Nephrol Dial Transplant. 2005;20(6):1038–41. PMID: 15840678. DOI: 10.1093/ndt/gfh824. [PubMed: 15840678] [CrossRef]
50.
Patient-Centered Outcomes Research Institute. Patient-Centered Outcomes Research. https://www​.pcori.org​/research-results/patient-centered-outcomes-research. Accessed June 4, 2019.
51.
Patient-Centered Outcomes Research Institute. PCORI Methodology Standards. April 2018. https://www​.pcori.org​/sites/default/files​/PCORI-Methodology-Standards.pdf. Accessed June 4, 2019.
52.
Li Q, Glynn RJ, Dreyer NA, et al. Validity of claims-based definitions of left ventricular systolic dysfunction in Medicare patients. Pharmacoepidemiol Drug Saf. 2011;20(7):700–8. PMID: 21608070. DOI: 10.1002/pds.2146. [PubMed: 21608070] [CrossRef]
53.
U.S. Food and Drug Administration. National Evaluation System for Health Technology (NEST). https://nestcc​.org/. Accessed June 4, 2019.
54.
Hodgson DC, Fuchs CS, Ayanian JZ. Impact of patient and provider characteristics on the treatment and outcomes of colorectal cancer. J Natl Cancer Inst. 2001;93(7):501–15. PMID: 11287444. DOI: 10.1093/jnci/93.7.501. [PubMed: 11287444] [CrossRef]
55.
Reeves MJ, Fonarow GC, Zhao X, et al. Quality of care in women with ischemic stroke in the GWTG program. Stroke. 2009;40(4):1127–33. PMID: 19211482. DOI: 10.1161/STROKEAHA.108.543157. [PubMed: 19211482] [CrossRef]
56.
Fonarow GC, Abraham WT, Albert NM, et al. Influence of a performance-improvement initiative on quality of care for patients hospitalized with heart failure: results of the Organized Program to Initiate Lifesaving Treatment in Hospitalized Patients With Heart Failure (OPTIMIZE-HF). Arch Intern Med. 2007;167(14):1493–502. PMID: 17646603. DOI: 10.1001/archinte.167.14.1493. [PubMed: 17646603] [CrossRef]
57.
Greene FL, Gilkerson S, Tedder P, et al. The role of the hospital registry in achieving outcome benchmarks in cancer care. J Surg Oncol. 2009;99(8):497–9. PMID: 19466739. DOI: 10.1002/jso.21186. [PubMed: 19466739] [CrossRef]
58.
Schweikert B, Hunger M, Meisinger C, et al. Quality of life several years after myocardial infarction: comparing the MONICA/KORA registry to the general population. European heart journal. 2009;30(4):436–43. PMID: 19019995. DOI: 10.1093/eurheartj/ehn509. [PubMed: 19019995] [CrossRef]
59.
Griffiths I, Silman A, Symmons D, et al. BSR Biologics Registry. Rheumatology (Oxford). 2004;43(12):1463–4. PMID: 15328420. DOI: 10.1093/rheumatology/keh356. [PubMed: 15328420] [CrossRef]
60.
U.S. Food and Drug Administration. Postapproval Pregnancy Safety Studies. DRAFT Guidance for Industry. May 2019. https://www​.fda.gov/media​/124746/download. Accessed June 4, 2019.
61.
Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–81. PMID: 21142528. DOI: 10.1056/NEJMp1011024. [PubMed: 21142528] [CrossRef]
62.
Connock M, Burls A, Frew E, et al. The clinical effectiveness and cost-effectiveness of enzyme replacement therapy for Gaucher’s disease: a systematic review. Health Technol Assess. 2006;10(24):iii–iv, ix–136. PMID: 16796930. [PubMed: 16796930]
63.
Connock M, Juarez-Garcia A, Frew E, et al. A systematic review of the clinical effectiveness and cost-effectiveness of enzyme replacement therapies for Fabry’s disease and mucopolysaccharidosis type 1. Health Technol Assess. 2006;10(20):iii–iv, ix–113. PMID: 16729919. [PubMed: 16729919]
64.
Chalkidou K, Tunis S, Lopert R, et al. Comparative effectiveness research and evidence-based health policy: experience from four countries. Milbank Q. 2009;87(2):339–67. PMID: 19523121. DOI: 10.1111/j.1468-0009.2009.00560.x. [PMC free article: PMC2881450] [PubMed: 19523121] [CrossRef]
65.
Parikh RB, Kakad M, Bates DW. Integrating Predictive Analytics Into High-Value Care: The Dawn of Precision Delivery. JAMA. 2016;315(7):651–2. PMID: 26881365. DOI: 10.1001/jama.2015.19417. [PubMed: 26881365] [CrossRef]
66.
Graves SE, Davidson D, Ingerson L, et al. The Australian Orthopaedic Association National Joint Replacement Registry. Med J Aust. 2004;180:(5 Suppl):S31–4. PMID: 14984361. [PubMed: 14984361]
67.
Owen A, Spinks J, Meehan A, et al. A new model to evaluate the long-term cost effectiveness of orphan and highly specialised drugs following listing on the Australian Pharmaceutical Benefits Scheme: the Bosentan Patient Registry. J Med Econ. 2008;11(2):235–43. PMID: 19450082. DOI: 10.3111/13696990802034525. [PubMed: 19450082] [CrossRef]
68.
Matchar DB, Oddone EZ, McCrory DC, et al. Influence of projected complication rates on estimated appropriate use rates for carotid endarterectomy. Appropriateness Project Investigators. Academic Medical Center Consortium. Health Serv Res. 1997;32(3):325–42. PMID: 9240284. [PMC free article: PMC1070194] [PubMed: 9240284]
69.
National Academies of Sciences, Engineering, and Medicine. Health and Medicine Division. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. September 2012. http://www​.nationalacademies​.org/hmd/Reports​/2012/Best-Care-at-Lower-Cost-The-Path-to-Continuously-Learning-Health-Care-in-America.aspx. Accessed June 4, 2019. [PubMed: 24901184]
©2020 United States Government, as represented by the Secretary of the Department of Health and Human Services, by assignment.

All rights reserved. The Agency for Healthcare Research and Quality (AHRQ) permits members of the public to reproduce, redistribute, publicly display, and incorporate this work into other materials provided that it must be reproduced without any changes to the work or portions thereof, except as permitted as fair use under the U.S. Copyright Act. This work contains certain tables and figures noted herein that are subject to copyright by third parties. These tables and figures may not be reproduced, redistributed, or incorporated into other materials independent of this work without permission of the third-party copyright owner(s). This work may not be reproduced, reprinted, or redistributed for a fee, nor may the work be sold for profit or incorporated into a profit-making venture without the express written consent of AHRQ. This work is subject to the restrictions of Section 1140 of the Social Security Act, 42 U.S.C. § 1320b-10. When parts of this work are used or quoted, the following citation should be used:

Bookshelf ID: NBK562581

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (4.0M)

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...