U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Gliklich RE, Leavy MB, Dreyer NA, editors. Registries for Evaluating Patient Outcomes: A User’s Guide [Internet]. 4th edition. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020 Sep.

Cover of Registries for Evaluating Patient Outcomes: A User’s Guide

Registries for Evaluating Patient Outcomes: A User’s Guide [Internet]. 4th edition.

Show details

Chapter 3Registry Design

1. Introduction

This chapter is intended as a high-level practical guide to the application of epidemiologic methods that are particularly useful in the design of registries that evaluate patient outcomes. Since it is not intended to replace a basic textbook on epidemiologic design, readers are encouraged to seek more information from textbooks and scientific articles. Table 3-1 summarizes the key considerations for study design that are discussed in this chapter. Throughout the design process, registry planners may want to discuss options and decisions with the registry stakeholders and relevant experts to ensure that sound decisions are made. The choice of groups to be consulted during the design phase generally depends on the nature of the registry, the registry funding source and funding mechanism, and the intended audience for registry reporting.

Table 3-1. Considerations for study design.

Table 3-1

Considerations for study design.

2. Research Questions Appropriate for Registries

The questions typically addressed in registries range from purely descriptive questions aimed at understanding the characteristics of people who develop the disease and how the disease generally progresses, to highly focused questions intended to support decision making. Registries focused on determining clinical effectiveness or cost-effectiveness or assessing safety or harm are generally hypothesis driven and concentrate on evaluating the effects of specific treatments on patient outcomes. Research questions should address the registry’s purposes, as broadly described in Table 3-2.

Table 3-2. Overview of registry purposes.

Table 3-2

Overview of registry purposes.

Observational studies derived from registries (or “registry-based studies”) are an important part of the research armamentarium alongside interventional studies, such as randomized controlled trials (RCTs), registry-based randomized trials, or other pragmatic randomized trials, and retrospective studies, such as studies derived exclusively from administrative claims data. Each of these study designs has strengths and limitations, and the selection of a study design should be guided by the research questions of interest. (See Chapter 2 for a discussion of the factors that influence the study design decision.) In some cases, multiple studies with different designs or a hybrid study that combines study designs will be necessary to address a research question. In fact, this more comprehensive approach to evidence development is likely to become more common as researchers strive to address multiple questions for multiple stakeholders most efficiently. Observational studies and interventional studies are more complementary than competitive, precisely because some research questions are better answered by one method than the other. Interventional studies are considered by many to provide the highest grade evidence for evaluating whether a drug has the ability to bring about an intended effect in optimal or “ideal world” situations, a concept also known as “efficacy.”1 Observational designs, on the other hand, are particularly well suited for studying broader populations, understanding actual results (e.g., some safety outcomes) in real-world practice, and for obtaining more representative quality-of-life information. This is particularly true when the factors surrounding the decision to treat are an important aspect of understanding treatment effectiveness.2

In many situations, nonrandomized comparisons either are sufficient to address the research question or, in some cases, may be necessary because of the following issues with randomizing patients to a specific treatment:

  • Equipoise: Can providers ethically introduce randomization between treatments when the treatments may not be clinically equivalent?
  • Ethics: If reasonable suspicion about the safety of a product has become known, would it be ethical to conduct a trial that deliberately exposes patients to potential harm? For example, can pregnant women be ethically exposed to drugs that may be teratogenic?
  • Practicality: Will patients enroll in a study where they might not receive the treatment, or might not receive what is perceived to be the best treatment? How can adherence to a treatment be studied, if not by observing what people do in real-world situations?

Registries are particularly suitable for some types of research questions, such as:

  • Natural history studies where the goal is to observe clinical practice and patient experience but not to introduce any intervention.
  • Studies of rare diseases or rare exposures that often require working with many sites to study relatively few patients.
  • Measures of clinical effectiveness, especially as related to adherence, where the purpose is to learn about what patients and practitioners actually do and how their actions affect real-world outcomes. This is especially important for treatments that have poor adherence.
  • Studies of effectiveness and safety for which clinician training and technique are part of the study of the treatment (e.g., a procedure such as placement of carotid stent).
  • Studies of heterogeneous patient populations, since unlike randomized trials, registries generally have much broader inclusion criteria and fewer exclusion criteria. These characteristics lead to studies with greater generalizability (external validity) and may allow for assessment of subgroup differences in treatment effects.
  • Followup for delayed or long-term benefits or harm, since registries can extend over much longer periods than most clinical trials (because of their generally lower operational costs and lesser burden on participants).
  • Surveillance for rare events.
  • Studies for treatments in which randomization is unethical, such as intentional exposure to potential harm (as in safety studies of marketed products that are suspected of being harmful).
  • Studies for treatments in which randomization is not necessary, such as when certain therapies are only available in certain places owing to high cost or other restrictions (e.g., proton beam therapy).
  • Studies for which blinding is challenging or unethical (e.g., studies of surgical interventions, complex or sequential treatments, acupuncture).
  • Studies of rapidly changing technology.
  • Studies of conditions with complex treatment patterns and treatment combinations.
  • Studies of healthcare access and barriers to care.
  • Evaluations of actual standard medical practice.
  • Studies of diagnostic outcomes, particularly when the outcome of interest is relatively rare and large cohorts are needed to assess test performance metrics.

Registry studies may also include embedded substudies as part of their overall design. These substudies can themselves have various designs (e.g., highly detailed prospective data collection on a subset of registry participants, or a case-control study focused on either incident or prevalent cases identified within the registry). Registries can also be used as a framework for RCTs.3,4

3. Translating Clinical Questions Into Measurable Exposures and Outcomes

The specific clinical questions of interest in a registry will guide the definitions of study subjects, exposure, and outcome measures, as well as the study design, data collection, and analysis. In the context of registries, the term “exposure” is used broadly to include treatments and procedures, healthcare services, diseases, and conditions.

The clinical questions of interest can be defined by reviewing published clinical information, soliciting experts’ opinions, and evaluating the expressed needs of the patients, healthcare providers, payers, and other stakeholders. Examples of research questions, key outcome and exposure variables, and sources of data are shown in Table 3-3.

Table 3-3. Examples of research questions and key exposures and outcomes.

Table 3-3

Examples of research questions and key exposures and outcomes.

As these examples show, the outcomes are the main endpoints of interest posed in the research question. These typically represent measures of health, onset or progression of illness, or adverse events; they also commonly include patient-reported outcome measures, such as quality of life measures, and measures of healthcare utilization and costs. More information on selecting outcome measures is provided in Chapter 4.

In addition to outcomes, relevant exposures also derive from the main research question and relate to why a patient might experience benefit or harm. Evaluation of an exposure includes collection of information that affects or augments the main exposure, such as dose, duration of exposure, route of exposure, or adherence. Other variables of interest include independent risk factors for the outcomes of interest (e.g., comorbidities, age), as well as variables known as potential confounding variables, that are related to both the exposure and the outcome and are necessary for conducting valid statistical analyses. Confounding can result in inaccurate estimates of association between the study exposure and outcome through mixing of effects. To continue with an asthma example, a study of a new asthma medication should collect prior history of treatment resistance or else results may be biased. The bias could occur because treatment resistance may relate both to the likelihood of receiving the new drug (meaning that doctors will be more likely to try a new drug in patients who have failed other therapies) and the likelihood of having a poorer outcome (e.g., hospitalization). Some efforts to standardize outcome measures, such as the OMF project, specify key risk factors and potential confounding variables that should be captured. Refer to Chapters 4 and 5 for more information.

4. Finding the Necessary Data

The identification of key outcome and exposure variables and patients will drive the strategy for data collection, including the choice of data sources. A key challenge to registries, as with all studies that require primary data collection, is that it may not be possible to collect all desired data. As discussed in Chapter 5, data collection should be both purpose-driven and broadly applicable. For example, while experimental imaging studies may provide interesting data, if the imaging technology is not widely available, the data will not be available for enough patients to be useful for analysis. Moreover, the registry findings will not be generalizable if only sophisticated centers that have such technology participate. Instead, registries should focus on collecting the data necessary to achieve their purpose(s) while minimizing the burden on patients and clinicians when feasible.

Registry data can be obtained from patients, clinicians, medical records, and linkage with other sources. While many registries relied on primary data collection in the past, the increasing availability of electronic healthcare data has introduced new opportunities for registries to capture data from secondary sources, such as electronic medical records and administrative databases, thus reducing data collection burden and increasing efficiency. More information on the technical aspects of linking or integrating existing data sources into registries can be found in the supplemental eBook on Registry Informatics.5 These approaches can yield rich datasets on large patient cohorts that can be used to address the primary objective of the registry as well as numerous secondary objectives. However, significant effort is often needed to clean, standardize, and normalize the data, and these data may not be recorded with the same rigor and quality assurance procedures that are used in some registries. Chapters 6 and 11 explore these issues.

5. Resources and Efficiency

Ideally, a study is designed to optimally answer a research question of interest and funded adequately to achieve the objectives based on the requirements of the design. Frequently, however, finite resources are available at the outset of a project that constrain the approaches that may be pursued. Often, through efficiencies in the selection of a study design and patient population (observational vs. RCT, case-control vs. prospective cohort), selection of data sources (e.g., use of secondary data sources vs. information collected directly from clinicians or patients), restriction of the number of study sites, or other approaches, studies may be planned that provide adequate evidence for addressing a research question within a specified budget. Section 6 below discusses how certain designs may be more efficient for addressing some research questions.

6. Study Designs for Registries

Registries provide a framework for various types of observational study designs. Typically, a registry is designed to support a specific study, but additional studies may be nested as substudies within the registry framework to address secondary objectives or questions that arise during the course of the registry. Additional data may need to be collected to facilitate examination of questions that arise. Before capturing new data elements, the steps outlined in Chapter 2, including assessing feasibility, considering the necessary scope and rigor, and evaluating the regulatory/ethical impact, should be undertaken.

The study models of case series, cohort, case-control, and case-cohort are commonly applied to registry data and are described briefly here. Other models are useful in some situations, but are not covered here. For example, case-crossover studies are efficient designs for studying the effects of intermittent exposures (e.g., use of erectile dysfunction drugs) on conditions with sudden onset. Another example is a pre and post study that enrolls sites prior to introduction of new technology, collects baseline data, and continues data collection after new technology is available. Registries may also provide a platform for pragmatic randomized trials.6,7 In a pragmatic trial, patients or providers may be randomized as to which intervention or quality improvement tools they use; the comparators are generally one or more other active treatments (generally referred to as standard of care) rather than placebos; and patients are observed without further intervention. Also, there has been recent interest in applying the concept of adaptive clinical trial design to registries. The U.S. Food and Drug Administration defines an adaptive design as “a clinical trial design that allows for prospectively planned modifications to one or more aspects of the design based on accumulating data from subjects in the trial.8 While many long-term registries are modified after initiation, the more formal aspects of adaptive trial design have yet to be applied regularly to registries and observational studies.

Determining what framework will be used to analyze the data is important in designing the registry and registry data collection procedures. Readers are encouraged to consult textbooks of epidemiology and pharmacoepidemiology for more information. Many of the references in Chapter 13 relate to study design and analysis.

6.1. Case Series Design

Using a registry population to develop case series is a straightforward application that does not require sophisticated analytics. Depending on the generalizability of the registry itself, case series drawn from the registry can be used to describe the characteristics to be used in comparison to other case series (e.g., from spontaneous adverse event reports). Self-controlled methods, including self-controlled case series, are a relatively new set of methods that lends itself well to registry analyses as it focuses on only those subjects who have experienced the event of interest and uses an internal comparison to derive the relative (not absolute) incidence of the event during the time the subject is “exposed” compared with the incidence during the time when they are “unexposed.”9 This design implicitly controls for all confounders that do not vary over the followup time (e.g., gender, genetics, geographic area), as the subject serves as his or her own control. The self-controlled case series design may also be very useful in those circumstances where a comparison group is not available. Self-controlled case series require that the probability of exposure is not affected by the occurrence of an outcome; in addition, for non-recurrent events, the method works only when the event risk is small and varies over the followup time. Derivative methods, grouped as self-controlled cohort methods, include observational screening,10 interrupted time series,11 and temporal pattern discovery.12 These methods compare the rate of events post-exposure with the rate of events pre-exposure among patients with at least one exposure. Registries that leverage secondary data sources, such as electronic health records, are well-suited for these methods because they typically capture data on most if not all patients at each participating site.

6.2. Cohort Design

Cohort studies follow over time a group of people who possess a characteristic, to see if individuals in the group develop a particular endpoint or outcome. The cohort design is used for descriptive studies as well as for studies seeking to evaluate comparative effectiveness and/or safety or quality of care. Cohort studies may include only people with exposures (such as to a particular drug or class of drugs) or disease of interest. Cohort studies may also include one or more comparison groups for which data are collected using the same methods during the same period. A single cohort study may in fact include multiple cohorts, each defined by a common disease, characteristic, or exposure. Cohorts may be small, such as those focused on rare diseases, but often they target large groups of people (e.g., in safety studies), such as all users of a particular drug or device. Limitations of registry-based cohort studies may include lack of data on treatments provided outside the participating sites (e.g., a surgical registry may have limited information on the patient’s use of chiropractic treatments) and underreporting of outcomes if a patient leaves the registry or is not adequately followed up.13 These pitfalls should be considered and addressed when planning a study.

6.3. Case-Control Design

A case-control study gathers patients who have a particular outcome or exposure or who have suffered an adverse event (“cases”) and “controls” who have not but are representative of the source population from which the cases arise.14 If properly designed and conducted, it should yield results similar to those expected from a cohort study of the population from which the cases were derived. The case-control design is often employed for understanding the etiology of rare diseases15 because of its efficiency. In studies where expensive data collection is required, such as some genetic analyses or other sophisticated testing, the case-control design is more efficient and cost effective than a cohort study because a case-control design collects information only from cases and a sample of non-cases. However, if no de novo data collection is required, the use of the cohort design may be preferable since it avoids the challenge of selecting a suitable control group and the concomitant danger of introducing more bias.

Depending on the outcome, exposure, or event of interest, cases and controls may be identifiable within a single registry. For example, in the evaluation of restenosis after coronary angioplasty in patients with end-stage renal disease, investigators identified both cases and controls from an institutional percutaneous transluminal coronary angioplasty registry; in this example, controls were randomly selected from the registry and matched by age and gender.16 Alternatively, cases can be identified in the registry and controls chosen from outside the registry. Care must be taken, however, that the controls from outside the registry meet the requirement of arising from the same source population as the cases to which they will be compared. Matching in case-control designs—for example, ensuring that patient characteristics such as age and gender are similar in the cases and their controls—may yield additional efficiency, in that a smaller number of subjects may be required to answer the study question with a given power. However, matching does not eliminate confounding and must be undertaken with care. Matching variables must be accounted for in the analysis, because a form of selection bias similar to confounding will have been introduced by the matching.17

Properly executed, a case-control study can add efficiency to a registry if more extensive data are collected by the registry only for the smaller number of subjects selected for the case-control study. This design is sometimes referred to as a “nested” case-control study, since subjects are taken from a larger cohort. It is generally applied because of budgetary or logistical concerns relating to the additional data desired. Nested case-control studies have been conducted in a wide range of patient registries, from studying the association between oral contraceptives and various types of cancer using the Surveillance Epidemiology and End Results (SEER) program1820 to evaluating the possible association of depression with Alzheimer’s disease. As an example, in the latter case-control study design, probable cases were enrolled from an Alzheimer’s disease registry and compared with randomly selected nondemented controls from the same base population.21 The increasing availability of electronic healthcare data may make case-control designs unnecessary in some situations, as registries may be able to capture large volumes of data on large numbers of patient efficiently and use advanced statistical techniques, such as propensity score matching, to build cohorts for analysis. This approach is feasible when all necessary data are available in secondary sources, such as electronic health records. In cases where some data are unavailable in the medical record (e.g., patient-reported outcomes), case-control designs may be an appropriate option.

Case-control studies present special challenges with regard to control selection. More information on considerations and strategies can be found in a set of papers by Wacholder.2224

6.4. Case-Cohort Design

The case-cohort design is a variant of the case-control study. As in a case-control study, a case-cohort study enrolls patients who have a particular outcome or who have suffered an adverse event (“cases”), and “controls” who have not, but who are representative of the source population from which the cases arise. In nested case-control studies where controls are selected via risk-set sampling, each person in the source population has a probability of being selected as a control that is, ideally, in proportion to his or her person-time contribution to the cohort. In a case-cohort study, however, each control has an equal probability of being sampled from the source population.25 This allows for collection of pertinent data for cases and for a sample of the full cohort, instead of the whole cohort. For example, in a case-cohort study of histopathologic and microbiological indicators of chorioamnionitis, which included identification of specific microorganisms in the placenta, cases consisted of extreme preterm infants with cerebral palsy. Controls, which can be thought of as a randomly selected subcohort of subjects at risk of the event of interest, were selected from among all infants enrolled in a long-term study of preterm infants.26

With the assumptions that competing risks and loss to followup are not associated with the exposure or the risk of disease, the case-cohort design allows for the selection of one control group that can be compared with various case series since the controls are selected at the beginning of followup. Analogous to a cohort study where every subject in the source population is at risk for the disease at the start of followup, the control series in a case-cohort design represents a sample of the exposed and unexposed in the source population who are disease-free at the start of followup.

7. Choosing Patients for Study

The purpose of a registry is to provide information or describe events and patterns, and often to generate hypotheses about a specific patient population to whom study results are meant to apply. Studies can be conducted of people who share common characteristics, with or without the inclusion of comparison groups. For example, studies can be conducted of:

  • People with a particular disease/outcome or condition.
    • Examples include studies of the occurrence of cancer or rare diseases, pregnancy outcomes, and recruitment pools for clinical trials.
  • Those with a particular exposure (e.g., to a product, procedure, or other health service, or an environmental or personal exposure).
    • Examples include general surveillance registries, pregnancy registries for particular drug exposures, and studies of exposure to medications and to devices such as stents.27 They also include studies of people who were treated under a quality improvement program, studies of people with a specific environmental exposure28 or personal exposure29 and studies of a exposure that requires controlled distribution, such as drugs with serious safety concerns (e.g., isotretinoin, clozapine, natalizumab [Tysabri®]), where the participants in the registry are identified because of their participation in a controlled distribution/risk management program.
  • Those who were part of a program evaluation, disease management effort, or quality improvement project.
    • An example is the evaluation of the effectiveness of evidence-based program guidelines on improving treatment.

7.1. Target Population

Selecting patients for registries can be thought of as a multistage process that begins with understanding the target population (the population to which the findings are meant to apply, such as all patients with a disease or an exposure) and then selecting a sample of this population for study. Some registries will enroll all, or nearly all, of the target population, but most registries will enroll only a subset of the target population. The accessible study population is that portion of the target population to which the participating sites have access. The actual study population is the subset of those who can be identified, invited to participate, and who agree to participate.30 While it is desirable for the patients who participate in a study to be representative of the target population, representativeness is generally defined in terms of “typical” patients and providers, rather than through an actual sample of all known patients. It is rarely possible to enumerate the complete sample frame to facilitate statistical sampling, either for budgetary reasons or for reasons of practicality.31 An exception is registries composed of all users of a product (as in postmarketing surveillance studies where registry participation is required as a condition of receiving an intervention), an approach which is becoming more common to manage expensive interventions and/or to track potential safety issues.

Certain populations pose greater difficulties in assembling an actual study population that is truly representative of the target population. Children and other vulnerable populations present special challenges in recruitment, as they typically will have more restrictions imposed by institutional review boards and other oversight groups.

As with any research study, very clear definitions of the inclusion and exclusion criteria are necessary and should be well documented, including the rationale for these criteria. A common feature of registries is that they typically have few inclusion and exclusion criteria, which enhances their applicability to broader populations. Restriction, the strategy of limiting eligibility for entry to individuals within a certain range of values for a confounding factor, such as age, may be considered in order to reduce the effect of a confounding factor when it cannot otherwise be controlled, but this strategy may reduce the generalizability of results to other patients.

These criteria will largely be driven by the study objectives and any sampling strategy. For a more detailed description of target populations and their subpopulations, and how these choices affect generalizability and interpretation, see Chapter 13.

Once the patient population has been identified, attention shifts to selecting the institutions and providers from which patients will be selected. For more information on recruiting patients and providers, see Chapter 10. Depending on the purpose of the registry, direct enrollment of patients may also be appropriate. (See Case Examples 7, 12, and 20.)

7.2. Comparison Groups

Once the target population has been selected and the mechanism for their identification (e.g., by providers) is decided, the next decision involves determining whether to collect data on comparators. Depending on the purpose of the registry, internal or external (contemporaneous or historical) groups can be used to strengthen the understanding of whether the observed effects are different from what would be expected to occur. Comparison groups are most useful in registries where it is important to distinguish between alternative decisions or to assess relative benefits and risks of various treatments. Registries without comparison groups can be used for descriptive purposes, such as characterizing the natural history of a disease or condition, or for exploratory purposes and are often complemented with external benchmarks from contemporaneous or historical data. The addition of a comparison group may add significant complexity, time, and cost to a registry, although the cost can be quite modest if existing data can be used for selected points of comparison.

Although it may be appealing to use more than one comparison group in an effort to overcome the limitations that may result from using any single group, multiple comparison groups pose their own challenges to the interpretation of registry results. For example, the results of comparative safety and effectiveness evaluations may differ depending on the comparison group used, making interpretation of the findings difficult. Generally, it is preferable to make judgments about the best comparison group for study during the design phase and then concentrate resources on these subjects. Also, sensitivity analyses can be used to quantify the likely impact of bias on the study findings (See Chapter 13.)

The choice of comparison groups is more complex in registries than in randomized clinical trials. Whereas clinical trials use randomization to try to achieve an equal distribution of known and unknown risk factors between treatment groups, registry studies need to use various design and analytic strategies to adjust for the confounders that they have measured. The concern for any observational studies is that a person who receives a new intervention has different characteristics than people who receive other treatments or who receive no treatment at all, and that these different characteristics influence the person’s likelihood of experiencing benefit or harm. In other words, treatment choices are often related to demographic and lifestyle characteristics, stage of disease, and the presence of coexisting conditions that affect clinician decision making about treatments.32 To achieve comparability, registries may use inclusion criteria that, for example, restrict the registry focus to patients who have had the disease for a similar duration, are receiving their first treatment for a new condition, or are progressing to second-line treatments.

Other design techniques that can be used in registries, particularly those with large numbers of patients, include matching study subjects using statistical techniques (e.g., propensity scoring) to create strata of patients with similar likelihood of receiving a treatment or of experiencing benefits or risks. As an example, consider a recent study of a rare side effect in coronary artery surgery for patients with acute coronary syndrome. In this instance, the main exposure of interest was the use of antifibrinolytic agents during revascularization surgery, a practice that had become standard for such surgeries. The sickest patients, who were most likely to have adverse events, were much less likely to be treated with antifibrinolytic agents. To address this, the investigators measured more than 200 covariates (by drug and outcome) per patient and used this information in a propensity score analysis. The results of this large-scale observational study revealed that the traditionally accepted practice (aprotinin) was associated with serious end-organ damage and that the less expensive generic medications were safe alternatives.33 Registries that capture large volumes of data from secondary sources are particularly well-suited for propensity score analysis because the large amounts of data can produce better models and more comparable cohorts. The drawback to post-hoc matching is the loss of information since some patients in the treatment (or disease) group will be excluded from the analysis if a suitable match is not available; there are also debates concerning the impact of how groups are weighted when propensity scores are used (see Chapter 13.)34,35

An internal comparison group refers to simultaneous data collection for patients who are similar to the focus of interest (i.e., those with a particular disease or stage of disease), but who do not have the condition or exposure of interest. For example, a registry might collect information on patients with arthritis who are using acetaminophen for pain control. An internal comparison group could be arthritis patients who are using other medications for pain control. Data regarding similar patients, collected during the same calendar period and using the same data collection methods, are also useful for characterizing treatment heterogeneity and for understanding various risk factors, such as for studying the effects in certain age categories or among people with similar comorbidities. However, the information value and utility of these comparisons depend largely on having adequate numbers of patients in the subgroups of interest, and such analyses may need to be specified a priori to ensure that recruitment supports them. Internal comparisons are particularly useful because data are collected using the same tools and endpoints and during the same observation period as for all study subjects, which will account for time-related influences that may be external to the study while also assuring the requisite data are available for all study subjects. For example, if an important scientific article is published that affects general clinical practice, and the publication occurs during the period in which the study is being conducted, clinical practice may change. The effects may be comparable for groups observed during the same period through the same system, whereas information from historical comparisons, for example, would be expected to reflect different practices.

An external comparison group is a group of patients similar to those who are the focus of interest, but who do not have the condition or exposure of interest, and for whom relevant data have been collected outside of the registry. For example, the SEER program maintains national data about cancer and has provided useful comparison information for many registries where cancer is an outcome of interest.36 External comparison groups can provide informative benchmarks for understanding effects observed and for assessing generalizability, and they are currently being used by regulators in the United States and European Union to study treatment effectiveness for label expansions. In some cases, registry data are being used as the source of external comparators for phase II trials.37 Additionally, large clinical and administrative claims databases can contribute useful information on comparable subjects for a relatively low cost. Depending on the outcome of interest, a limitation of external comparison groups is that the data are generally not collected the same way and the same information may not be available; however, these differences may not be problematic for some outcomes, such as mortality.

External comparators may be contemporaneous (i.e., referring to data collected during the same timeframe as the registry patients) or historical, referring to patients who are similar to the focus of interest, but who do not have the condition or exposure of interest, and for whom information was collected in the past (such as before the introduction of an exposure or treatment or development of a condition). Historical controls may actually be the same patients who later become exposed, or they may consist of a completely different group of patients. For example, historical comparators are often used for pregnancy studies since there is a large body of population-based surveillance data available, such as the Metropolitan Atlanta Congenital Defects Program (MACDP).38 This design provides weak evidence because symmetry is not assured (i.e., the patients in different time periods may not be as similar as desired). Historical controls are susceptible to bias by changes over time in uncontrollable, confounding risk factors, such as differences in climate, management practices, and nutrition. Bias stemming from differences in measuring procedures over time may also account for observed differences.

An approach related to the use of historical comparisons is the use of Objective Performance Criterion (OPC) as a comparator. This research method has been described as an alternative to randomized trials, particularly for the study of devices.39 An OPC “refers to a numerical target value derived from historical data from clinical studies and/or registries and may be used in a dichotomous (pass/fail) manner by FDA for the review and comparison of safety or effectiveness endpoints.”40 A U.S. Food and Drug Administration guidance document for pivotal clinical investigations for medical devices includes a description of how OPCs may be used in the context of medical device studies. Registries serve as a source of reliable historical data in this context, particularly when combined with trials and other data sources.

There are several situations in which internal comparators may be impractical, unethical, or impossible and a historical comparison may be considered:

  • When one cannot ethically continue the use of older treatments or practices, or when clinicians and/or patients refuse to continue their use, so that the researcher cannot identify relevant sites using the older treatments.
  • When uptake of a new medical practice has been rapid, concurrent comparisons may differ so markedly from treated patients, with regard to factors related to outcomes of interest, that they cannot serve as valid comparison subjects due to intractable confounding.
  • When conventional treatment has been consistently unsuccessful and the effect of new intervention is obvious and dramatic (e.g., first use of a new product for a previously untreatable condition).
  • When collecting the comparison data is too expensive.
  • When the Hawthorne effect (a phenomenon that refers to changes in the behavior of subjects because they know they are being studied or observed) makes it impossible to replicate actual practice in a comparison group during the same period.
  • When the desired comparison is to usual care or “expected” outcomes at a population level, and data collection is too expensive due to the distribution or size of that population.

8. Registry Size and Duration

Precision in measurement and estimation corresponds to the reduction of random error. Depending on available budget, precision can sometimes be improved by increasing the number of study subjects or followup period.41

During the registry design stage, it is critical to explicitly state the target number of sites and patients, how long patients should be followed, and the justifications for these decisions. These decisions should be based on the overall purpose of the registry but tempered by budget and whether the proposed registry would fill an important information gap, even if the target study size is not optimal. For example, in addressing specific questions of product safety or effectiveness, the desired level of precision to confirm or rule out the existence of an important effect should be specified, and ideally should be linked to policy or practice decisions that will be made based on the evidence. Nonetheless, registries may make important contributions even if they are only able to evaluate large effects, should they exist.42 For registries with aims that are descriptive or hypothesis generating, study size may be arrived at through other considerations.

The duration of registry enrollment and followup should be determined both by required number of patients or person-years desired to achieve the target statistical power and by time- and budget-related considerations. The expected (or theoretical) induction period for some outcomes of interest should be considered, and ideally, sufficient followup time allowed for the exposure under study to have induced or promoted the outcome. Calendar time may be a consideration in studies of changes in clinical practice or interventions that have a clear beginning and end. The need for evidence to inform policy may also determine a timeframe within which the evidence must be made available to decision-makers. For practical purposes, it may be useful to evaluate the risk over a time period that is feasible to study, for example, characterizing the benefits and risks for five years after receipt of an artificial hip, recognizing that a much longer time period may also be of interest, e.g., how does the hip perform after 10 years.

A detailed discussion of the topic of study size calculations for registries is provided in Appendix A. For present purposes it is sufficient to briefly describe some of the critical inputs to these calculations that must be provided by the registry developers:

  • The expected timeframe of the registry and the time intervals at which analyses of registry data will be performed.
  • Either the expected size of clinically meaningful effects (e.g., minimum clinically important differences) or the desired precision of the effect estimates.
  • Whether or not the registry is intended to support regulatory decision making. If the results from the registry will affect regulatory action—for example, the likelihood that a product may be pulled from the market—then the precision of the overall risk estimate is important, as is the necessity to predict and account for attrition in designing the target study size.

In a classical calculation of sample size, the crucial inputs that must be provided by the investigators include either the size of clinically important effects or their required precision. For example, suppose that the primary goal of the registry is to compare surgical complication rates in general practice with those in randomized trials. The inputs to the statistical power calculations would include the complication rates from the randomized trials (e.g., 4 percent) and the complication rate in general practice, which would reflect a meaningful departure from this rate (e.g., 6 percent). If, on the other hand, the goal of the registry is simply to track complication rates (and not to compare the registry with an external standard), then the investigators should specify the required width of the confidence interval associated with those rates. For example, in a large registry, the 95-percent confidence interval for a 5-percent complication rate might extend from 4.5 percent to 5.5 percent. If all of the points in this confidence interval lead to the same decision, then an interval of ±0.5 percent is considered sufficiently precise, and this is the input required for the estimation of sample size.

Specifying the above inputs to sample size calculations is a substantial matter and usually involves a combination of quantitative and qualitative reasoning. The issues involved in making this specification are essentially similar for registries and other study designs, though for registries designed to address multiple questions of interest, one or more primary objectives or endpoints must be selected that will drive the selection of a minimum sample size to meet those objectives.

Other considerations that may be taken into account when estimating study sizes include—

  • whether multiple comparisons are being made and subjected to statistical testing, although this notion has been soundly challenged;43 and
  • whether levels of expected attrition or lack of adherence to therapy may require a larger number of patients to achieve the desired number of person-years of followup or exposure.

Although most of the emphasis in estimating study size requirements is focused on patients, it is equally important to consider the number of sites needed to recruit and retain enough patients to achieve a reasonably informative number of person-years for analysis. Many factors are involved in choosing the number and types of sites needed for a given study, including the number of eligible patients seen in a given practice during the relevant time period, desired representativeness of sites with regard to geography, practice size, or other features, and the timeframe within which study results are required, which may also limit the timeframe for patient recruitment.

In summary, the aims of a registry, the desired precision of information sought, and the research question(s) determine the process and inputs for arriving at a target sample size and specifying the duration of followup.

Registries with mainly descriptive aims, or those that provide quality metrics for clinicians or medical centers, may not require the choice of a target study size to be arrived at through statistical power calculations. In these cases, the costs of obtaining study data, in monetary terms and in terms of researcher, clinician, and patient time and effort, may set upper as well as lower limits on study size and scope.

9. Internal and External Validity

The potential for bias refers to opportunities for systematic errors to influence the results. Internal validity is the extent to which study results are free from bias, and the reported association between exposure and outcome is not due to unmeasured or uncontrolled-for variables. Generalizability, also known as external validity, is a concept that refers to the utility of the inferences for the broader population that the study subjects are intended to represent. In considering potential biases and generalizability, we discuss the differences between RCTs and registries, since these are the two principal approaches to conducting clinically relevant prospective research.

The strong internal validity that earns RCTs high grades for evidence comes largely from the randomization of exposures that helps ensure that the groups receiving the different treatments are similar in all measured or unmeasured characteristics, and that, therefore, any differences in outcome (beyond those attributable to chance) can be likely attributed to differences in the efficacy or safety of the treatments. It should be noted that randomization does not guarantee perfect balancing of risk factors and that RCTs are not without their own biases, as illustrated by the “intent-to-treat” analytic approach, in which people are considered to have used the assigned treatment, regardless of actual adherence. The intent-to-treat analyses can minimize a real difference—generating a distortion known as “bias toward the null”—by including the experience of people who did not adhere to the recommended study product along with those who did.

Another principal difference between registries and RCTs is that RCTs generally focus on a relatively homogeneous pool of patients from which significant numbers of patients are purposefully excluded at the cost of external validity—that is, generalizability to the target population of disease sufferers. Registries, in contrast, usually focus on generalizability so that their population will be representative and relevant to decision-makers.

9.1. Generalizability

The strong external validity of registries is achieved by the fact that they include typical patients, which often include more heterogeneous populations than those participating in RCTs (e.g., wide variety of age, ethnicity, and comorbidities). Therefore, registry data can provide a good description of the course of disease and impact of interventions in actual practice. For many purposes, registries may be more relevant for decision making than the data derived from the artificial constructs of the clinical trial because registries generally represent more diverse (and more typical) medical practice as well as more diverse patients. In fact, even though registries have more opportunities to introduce bias (systematic error) because of their nonexperimental methodology, well designed observational studies can approximate the effects of interventions observed in RCTs on the same topic44,45 and, in particular, in the evaluation of healthcare effectiveness in many instances,46 and can provide information that may be more relevant to typical clinical practice.

The choice of groups from which patients will be selected directly affects generalizability. No particular method will ensure that an approach to patient recruitment is adequate, but it is worthwhile to note that the way in which patients are recruited, classified, and followed can either enhance or diminish the external validity of a registry. Some examples of how these methods of patient recruitment and followup can lead to systematic error follow.

9.2. Information Bias

If the registry’s principal goal is the estimation of risk, it is possible that adverse events or the number of patients experiencing them will be underreported if the reporter will be viewed negatively for reporting them. It is also possible for those collecting data to introduce bias by misreporting the outcome of an intervention if they have a vested interest in doing so. This type of bias is referred to as information bias (also called detection, observer, ascertainment, or assessment bias), and it addresses the extent to which the data that are collected are valid (represent what they are intended to represent) and accurate. This bias arises if the outcome assessment can be interfered with, intentionally or unintentionally. On the other hand, if the outcome is objective, such as whether or not a patient died or the results of a lab test, then the data are unlikely to be biased.

9.3. Selection Bias

A registry may create the incentive to enroll only patients who either are at low risk of complications or who are known not to have suffered such complications, biasing the results of the registry toward lower event rates. For example, a registry designed to assess complication rates that enrolls hospitals or surgeons who would derive benefit from reporting low complication rates would be at particularly high risk for this type of bias. Another example of how patient selection methods can lead to bias is the use of patient volunteers, a practice that may lead to selective participation from subjects most likely to perceive a benefit, distorting results for studies of patient-reported outcomes.

Enrolling patients who share a common exposure history, such as having used a drug that has been publicly linked to a serious adverse effect, could distort effect estimates for cohort and case-control analyses. Registries can also selectively enroll people who are at higher risk of developing serious side effects, since having a high-risk profile can motivate a patient to participate in a registry.

The term selection bias refers to situations where the procedures used to select study subjects lead to an effect estimate among those participating in the study that is different from the estimate that is obtainable from the target population.47 Selection bias may be introduced if certain subgroups of patients are routinely included or excluded from the registry. Selection bias also may arise when patients must provide informed consent to participate in the registry. Some research has shown that patients who consent to participate in clinical research are different from patients who elect not to participate.48 Depending on the registry purpose and design, some registries may be able to obtain a waiver of informed consent from an institutional review board; in these cases, data on all eligible patients are obtained, thus avoiding the potential for bias related to enrollment procedures.

9.4. Channeling Bias (Confounding by Indication)

Channeling bias, also called confounding by indication, is a form of selection bias in which drugs with similar therapeutic indications are prescribed to groups of patients with prognostic differences.49 For example, physicians may prescribe new treatments more often to those patients who have failed on traditional first-line treatments.

One approach to designing studies to address channeling bias is to conduct a prospective review of cases, in which external reviewers are blinded as to the treatments that were employed and are asked to determine whether a particular type of therapy is indicated and to rate the overall prognosis for the patient.50 This method of blinded prospective review was developed to support research on ruptured cerebral aneurysms, a rare and serious situation. The results of the blinded review were used to create risk strata for analysis so that comparisons could be conducted only for candidates for whom both therapies under study were indicated, a procedure much like the application of additional inclusion and exclusion criteria in a clinical trial.

For registries with sufficient data, statistical approaches such as matching subjects using propensity scores (i.e., the predicted probability of use of one therapy over another based on medical history, healthcare utilization, and other characteristics measured prior to the initiation of therapy) may be incorporated into study designs to address this type of confounding.51,52 Propensity scores may be used to create cohorts of initiators of two different treatments matched with respect to probability of use of one of the two therapies, for stratification or for inclusion as a covariate in a multivariate analysis. Studies incorporating propensity scores as part of their design may be planned prior to and implemented shortly following launch of a new drug as part of a risk management program, with matched comparators being selected over time, so that differences in prescribing patterns following drug launch may be taken into account.53

Registries with large amounts of data, such as quality improvement registries, may also consider instrumental variables, or factors strongly associated with treatment but related to outcome only through their association with treatment, as an additional means of adjustment for confounding by indication, as well as unmeasured confounding.54 Types of instrumental variables include providers’ preferences for one therapy over another—a variable which exploits variation in practice as a type of natural experiment; variation or changes in insurance coverage or economic factors (e.g., cigarette taxes) associated with an exposure; or geographic distance from a specific type of service.5558 Variables that serve as effective instruments of this nature are not always available and may be difficult to identify. While use of clinician or study site may, in some specific cases, offer potential as an instrumental variable for analysis, the requirement that use of one therapy over another be strongly associated with the instrument is often difficult to meet in real-world settings. That said, instrumental variable analysis may either support the conclusions drawn on the basis of the initial analysis, or it may raise additional questions regarding the potential impact of confounding by indication.51

In some cases, however, differences in disease severity or prognosis between patients receiving one therapy rather than another may be so extreme and/or unmeasurable that confounding by indication is not remediable in an observational design that compares one group to another.59 This represents special challenges for observational studies of comparative effectiveness, as the severity of underlying illness may be a strong determinant of both choice of treatment and treatment outcome.

9.5. Bias From Study of Existing Rather Than New Product Users

If there is any potential for tolerance to affect the use of a product, such that only those who perceive benefit from it or are free from harm continue using it, the recruitment of existing users rather than new product users may lead to the inclusion of only those who have tolerated or benefited from the intervention, and would not necessarily capture the full spectrum of experience and outcomes. This approach is generally used with pharmacotherapy but is not as widely applicable to medical devices studies, since prior use of a medical device may not influence a patient’s likelihood of tolerating it again (e.g., a patient’s experience with right knee replacement may not predict experience with left knee replacement).60 Selecting only existing users may introduce any number of biases, including incidence/prevalence bias, survivorship bias, and followup bias. By enrolling new users (an inception or incidence cohort), a study ensures that the longitudinal experience of all users will be captured, and that the ascertainment of their experience and outcomes will be comparable.61

9.6. Loss to Followup

Loss to followup or attrition of patients and sites threatens generalizability as well as internal validity if there is differential loss; for example, loss of participants with a particular exposure or disease, or with particular outcomes. Loss to followup and attrition are generally a serious concern only when they are nonrandom (that is, when there are systematic differences between those who leave or are lost and those who remain). The magnitude of loss to followup or attrition determines the potential impact of any bias. Given that the differences between patients who remain enrolled and those who are lost to followup are often unknown (unmeasurable), preventing loss to followup in long-term studies to the fullest extent possible will increase the credibility and validity of the results.62 Attrition should be considered with regard to both patients and study sites, as results may be biased or less generalizable if only some sites (e.g., teaching hospitals) remain in the study while others discontinue participation.

9.7. Assessing the Magnitude and Impact of Bias

Remaining alert for any source of bias is important, and the value of a registry is enhanced by its ability to provide a formal assessment of the likely magnitude of all potential sources of bias and their impact on the study findings. Any information that can be generated regarding nonrespondents, participants lost to followup, missing data and the like, is helpful, even if it is just an estimation of their raw numbers.

As with many types of survey research, an assessment of differential response rates and patient selection can sometimes be undertaken when key data elements are available for both registry enrollees and nonparticipants or drop-outs. Such analyses can easily be undertaken when the initial data source or population pool is that of a healthcare organization, employer, or practice that has access to data in addition to key selection criteria (e.g., demographic data or data on comorbidities); these types of analyses are more challenging when registries cross health systems, institutions, and borders.

Another tool is the use of sequential screening logs, in which all subjects fitting the inclusion criteria are enumerated and a few key data elements are recorded for all those who are screened to allow some quantitative analysis of nonparticipants and assessments of any differences in key characteristics or events. Whenever possible, quantitative assessment of the likely impact of bias is desirable to determine the sensitivity of the findings to varying assumptions. A text on quantitative analysis of bias through validation studies, and on probabilistic approaches to data analysis, provides a guide for planning and implementing these methods.63

Qualitative assessments, although not as rigorous as quantitative approaches, may give users of the research a framework for drawing some conclusions regarding the effects of bias on study results if the basis for the assessment is made explicit in reporting the results.

Accordingly, two items that can be reported to help the user assess the generalizability of research results based on registry data are a description of the criteria used to select the registry sites, and the characteristics of these sites, particularly those characteristics that might have an impact on the purpose of the registry. Consider, for example, a registry designed for the purpose of assessing adherence to lipid screening guidelines that requires sites to have a sophisticated electronic health record for data collection. In this scenario, adherence that is better than usual practice may be reported if the electronic medical record facilitates the generation of real-time reminders to engage in screening. Report of rates of adherence to other screening guidelines (for which there were no reminders), even if these are outside the direct scope of inquiry, would provide some insight into the degree of generalizability to other types of facilities.

Finally, and most importantly, whether or not study subjects need to be evaluated on their representativeness depends on the purpose and kind of inference needed. For example, sampling in proportion to the underlying distribution in the population is not necessary to understand biological effects. However, if the study purpose were to estimate a rate of occurrence of a particular event in a general population then sampling would be necessary to reflect the appropriate underlying distributions.

10. Special Considerations

The study design considerations discussed in this chapter apply to patient registries broadly. Some types of patient registries may need to consider additional factors when determining the most appropriate study design. The following sections summarize design considerations unique to registries designed for product safety assessment, rare diseases, and medical devices as well as pregnancy registries and quality improvement registries.

10.1. Designing Registries for Product Safety Assessmenti

Patient registries, particularly disease and product registries, that systematically collect data on all eligible patients are a tremendous resource for capturing important information about product safety. When designing a registry for safety, the size of the registry, the enrolled population, and the duration of followup are all critical to understanding the generalizability of results and applicability of the inferences made from the data. In addition, registries designed for safety must clearly define the exposure and risk windows under observation. The registry should record specific information about the products of interest, including route of administration, dose, duration of use, start and stop date, and, ideally, information about whether a generic or branded product was used (and which brand) and/or other pertinent information about the product. Studies of biologic medicines and devices benefit from including device identifiers, as well as information about production lots, and batches. Patterns of real-world product use, such as treatment switches, drug holidays, pill splitting and medication sharing, and patient non-adherence, should also be considered when designing the registry and during data collection. More information on designing registries for product safety assessment can be found in Chapter 19 of the third edition of the User’s Guide. Case Example 5 also provides a description of how a registry has provided data for product safety assessments.

10.2. Designing Registries for Medical Devicesii

Additional issues must be considered in the design phase of medical device registries to enable the registry to function across the lifecycle of device innovation. While drugs are typically identified through National Drug Codes (NDCs), identification of devices is more complex because of the iterative cycle of device modifications. In 2013, the FDA issued a final rule establishing a unique device identification system and requiring each device to have a Unique Device Identifier (UDI) on device labels and packages.64 A UDI is a unique alphanumeric or numeric code that contains a device identifier (describing the manufacturer and specific version or model of the device) and a production identifier (describing the lot or batch number, serial number, expiration date, manufactured date, or other distinct identification code). The FDA is requiring devices to have UDIs on a staged 7-year compliance schedule, ending in 2020. While devices are increasingly labeled with UDIs, capturing UDI data within a registry is still complicated. The FDA requirements only extend to the labeling of devices. Adoption and integration of UDIs into the healthcare delivery system is also necessary to facilitate capture of UDIs within registries. Routine and consistent capture of UDIs within electronic health records and administrative claims databases would facilitate re-use of these data for research purposes.

In addition to accurately identifying a device, medical device registries must consider how to capture factors related to device performance. These include potential performance issues, failure modes, and adverse events. The performance issues may be related to software, hardware, biomaterials, sterility, or other issues. In many cases, the device of interest for a registry is either part of a larger system of devices or contains multiple components that are considered devices themselves. Some implantable devices also require assistance from procedural devices, including other commodity devices or operative instruments, or ancillary devices, such as imaging equipment. Device/drug combinations, such as drug-eluting stents, have also become increasingly common in the past decade and necessitate separate collection of concomitant drug dosing information and attention to the medications that the patient is taking during and post implantation to flag possible drug interactions. When studying device safety or effectiveness, researchers should consider the role of these factors in device performance and how these data can be captured in the data collection process.

Lastly, provider experience and training can influence the selection of device, device performance, and patient outcomes, particularly for implantable devices. Device-specific training is an important element of a medical device registry that is not an issue in a drug registry, and experience-related factors such as practitioner annual volume, practitioner lifetime volume, facility volume, and facility characteristics such as academic teaching status should also be considered in analyses and training evaluations. It is ideal to have training and volume information in the registry, but this may not always be realistic. If this is deemed critical, information needs to be collected on provider experience and training at registry initiation and supplemented if any training programs occur during the registry development. More information on designing registries for medical devices can be found in Chapter 23 of the third edition of the User’s Guide.

10.3. Designing Registries for Rare Diseaseiii

Rare diseases present special research challenges due to the scarcity of relevant knowledge and experience. Prospective long-term patient registries are critical tools in building a broad and comprehensive knowledge base for these often heterogeneous diseases. Clinicians with relevant expertise who manage patients with rare diseases are limited, and a broad approach may be necessary to identify and recruit sufficient sites and patients to characterize the natural history of the disease. In some cases, multinational efforts may be necessary to enroll sufficient patients.

Registry design is also complicated by the absence of treatments or standards of care for many rare diseases. Use of experimental and adjunctive therapies is common, and it is often unclear how to characterize disease progression, especially start and stop dates of exacerbations. In addition to the typical objectives for disease registries (understand natural history and outcomes, assess effectiveness of treatments, etc.), rare disease registries may be designed to support the drug development process. In these cases, registries may be designed to recruit a readily available pool of patients for potential enrollment into clinical trials, gather baseline data that can inform trial design, and provide external reference groups for understanding potential treatment benefit and risks. Because of the scarcity of eligible patients, patient advocacy and support groups often play more active roles in rare disease registries than in more traditional disease registries;65 in some cases, these organizations may sponsor registries, or they may work with other sponsors as active partners in the development and operations of the registry. Lastly, the scope of rare disease registries frequently evolves over time, as understanding of the disease increases and/or new treatments become available. More information on designing registries for rare diseases can be found in Chapter 20 of the third edition of the User’s Guide.

10.4. Designing Pregnancy Registriesiv

Pregnancy registry design differs from the design of other types of registries in several respects. First, the population of a pregnancy registry can be defined based on women, pregnancies, or fetuses. A woman might have more than one pregnancy, and she might enroll in the same registry more than once. Clustered analyses are often used in this situation. In addition, multifetal gestations result in more than one fetus “enrolled” within the same pregnancy. Although there may be several ways of dealing with multiple gestations, it is prudent to collect information about all the fetuses. When reporting risks, whether using fetuses or pregnancies as the unit of analysis, both the numerator and denominator should be consistent with the choice. Registries should include women as soon as possible after conception, or even earlier at pregnancy planning stages, to allow the evaluation of early pregnancy events, and women should be enrolled before the pregnancy outcome is known to avoid a selection into the study affected by the outcome. An ideal pregnancy cohort would enroll women at conception and follow them for months beyond delivery, but, in practice, time from enrollment to end of followup can range from 1 month to over 1 year. As with any registry, longer followup periods lead to higher opportunities for diagnosis and therefore both larger cumulative risk estimates and greater statistical power. However, the length of followup may be influenced by the availability of resources and the registry’s ability to maintain contact with registry participants and/or healthcare providers over a longer term. It is also very difficult to enroll women early in pregnancy, and differential enrollment may occur according to whether it is a woman’s first pregnancy or she has been pregnant before.66

Necessary information on exposure, outcome, and key confounders (e.g., history, status, severity, and management of the indication) must be collected. Because treatment strategies often change during pregnancy, detailed information should be collected on treatment start and stop dates, dose, frequency, duration, and indication. Consideration should also be given to the source (mother, obstetrician, pediatrician) of information on outcomes and the potential for selective recall. A method for expert adjudication of birth defect classification, blinded to exposure status, is an important component of a pregnancy registry. In addition, the case of major birth defects occurring in pregnancies that end in embryonic or fetal demise must be considered in the registry design. Failure to include defects detected among terminations can decrease power and introduce bias, particularly for defects for which termination is often chosen after prenatal diagnosis (e.g., neural tube defects).

A critical element for pregnancy exposure registries is the choice of comparator groups. The most valid reference group will have comparable (1) outcome definition (e.g., exclusion of minor anomalies); (2) outcome assessment (e.g., intensity of screening, frequency of terminations, inclusion of prenatal diagnoses, availability of diagnostic tests, start and stop of followup);67 (3) selection of subjects into the study (e.g., gestational age at enrollment); and (4) baseline risk (e.g., distribution of risk factors, including indication). Ideally, each registry is constructed to include one or more internal reference groups, though this is not common practice. When this is not possible, an external reference group must be selected with care. Each comparison group has its advantages and disadvantages. For example, an external population-based reference group is generally larger and can provide more stable estimates for specific malformations, while an internal comparison group, which may be too small to support assessment of specific malformations, may be able to provide more comparable estimates for malformations overall. More than one comparison group can be used to enhance generalizability. More information on designing pregnancy registries can be found in Chapter 21 of the third edition of the User’s Guide. Case Example 6 also provides a description of a long-running pregnancy registry.

10.5. Designing Quality Improvement Registriesv

Designing a quality improvement (QI) registry presents several challenges, particularly when multiple stakeholders are involved. Like other types of registries, design of QI registries should be purpose-driven. This purpose may require detailed data collection at a single point in time (e.g., to improve care for patients hospitalized with acute coronary syndrome) or long-term followup data from different providers (e.g., to monitor care for patients with coronary artery disease). QI registries may focus on issues within a single institution, or they may address common treatment gaps that are relevant at many institutions.

A unique and critical component of QI registries are quality measures. Quality measures are tools that quantify healthcare processes or outcomes and are designed to help institutions and providers deliver high-quality care that aligns with clinical guidelines or best practices. Quality measures drive the registry data collection and reporting and thus form the backbone of a QI registry. Since QI registries are part of healthcare operations, it is critical that they do not overly interfere with the efficiency of those operations, and therefore the data collection must be limited to those data elements that are essential for calculating the relevant quality measures. In addition, the appropriate level of analysis and reporting of quality measures is an important consideration in QI registry design. Reports on compliance with quality measures may provide data at the individual patient, provider, or institution level, or they may provide aggregate data on groups of patients, providers, and institutions. The registry may also provide reports to the registry participants, to patients, or to the public. Reports may be unblinded (e.g., the provider is identifiable) or blinded, and they may be provided through the registry or through other means. In designing the registry, consideration should be given to what types of reports will be most relevant for achieving the registry’s goals, what types of reports will be acceptable to participants, and how those reports should be presented and delivered. More information on designing quality improvement registries can be found in Chapter 22 of the third edition of the User’s Guide. Case Examples 8 and 11 also describe quality improvement registries.

10.6. Designing Multinational Registries

In cases where the registry intends to collect data in more than one country, it is desirable to gather input from clinicians and patients in the other country (or countries) to understand potential variations in treatment patterns and data elements. Treatment patterns often vary across geographic regions due to multiple factors, including differences in approved indications, coverage decisions, and clinical guidelines. Products may be approved for different indications in different countries or regions, which can lead to the use of the product by patients with different characteristics, including varying levels of severity of conditions in each country or region. For example, natalizumab is approved in the European Union (EU) for patients who have failed two or more therapies for relapsing-remitting multiple sclerosis, while, in the United States, the therapy is used more widely. Differences in health insurance coverage decisions may affect treatment patterns in a similar manner; access and reimbursement levels may differ among countries, which can impact providers’ and patients’ ability and willingness to use a specific product. The use of different clinical guidelines also can have a substantial impact on treatment patterns. The American Gastroenterological Association, for example, recommends annual or biannual colonoscopic surveillance for neoplasia in patients with inflammatory bowel disease-related colitis, depending on whether patients are considered high risk or average risk. In contrast, the British Society of Gastroenterology recommends colonoscopic surveillance on a 1-year, 3-year, or 5-year basis, depending on risk assessment.68

Registries implemented in multiple countries must plan for these types of differences in standard of care. Because registries are observational, additional diagnostic or monitoring procedures such as laboratory tests are not undertaken unless they are within the scope of normal practice. Combined with differences in national guidelines, policies, and regulations, this makes variation in data availability commonplace for multinational registries.

11. Summary

In summary, the key points to consider in designing a registry include study design, selection of patients and healthcare practitioners, data collection, comparison groups, recruitment strategies, and considerations of possible sources of bias, their likely impact, and ways to address them to the extent that is practical and achievable. Additional design considerations apply for some specialized types of patient registries.

Lastly, it is important to keep in mind that it may be necessary to revisit the registry design if it becomes apparent that the initial plan will not meet expectations. For example, the original criteria for defining the target population (patients and/or healthcare providers) may not yield enough patients, such as when a treatment of interest is only slowly coming into use for the intended population or in the sites that have been recruited for study; moreover, recruitment can be more difficult if the treatment or product of interest is not covered by health insurers. More information on modifying patient registries can be found in Chapters 2 and 11.



Adapted from Blackburn S, Dreyer NA, Starzyk K. ‘Use of Registries in Product Safety Assessment.’ In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014 [PubMed: 24945055]


Adapted from Gross T, Kuntz RE, Mack C. ‘Medical Device Registries.’ In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014 [PubMed: 24945055]


Adapted from Dale DC, Groft SC, Harrison MJ, et al. ‘Rare Disease Registries.’ In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. [PubMed: 24945055]


Adapted from Chambers C, Covington D, Cragan JD, et al. ‘Pregnancy Registries.’ In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. [PubMed: 24945055]


Adapted from Asher AL, Gliklich RE, Hernandez AF, et al. ‘Quality Improvement Registries.’ In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. [PubMed: 24945055]

References for Chapter 3

Strom BL. Pharmacoepidemiology. 3rd ed. Chichester, England: John Wiley; 2000.
Gliklich R. A New Framework For Comprehensive Evidence Development. IN VIVO. October 22, 2012. https://invivo​.pharmaintelligence​.informa​.com/IV003921/A-New-Framework-For-Comprehensive-Evidence-Development. Accessed June 10, 2019.
James S, Rao SV, Granger CB. Registry-based randomized clinical trials--a new clinical trial paradigm. Nat Rev Cardiol. 2015;12(5):312–6. PMID: 25781411. DOI: 10.1038/nrcardio.2015.33. [PubMed: 25781411] [CrossRef]
Li G, Sajobi TT, Menon BK, et al. Registry-based randomized controlled trials- what are the advantages, challenges, and areas for future research? J Clin Epidemiol. 2016;80:16–24. PMID: 27555082. DOI: 10.1016/j.jclinepi.2016.08.003. [PubMed: 27555082] [CrossRef]
Gliklich RE, Leavy MB, Dreyer NA, (eds). Tools and Technologies for Registry Interoperability, Registries for Evaluating Patient Outcomes: A User’s Guide. 3rd Edition, Addendum II. (Prepared by L&M Policy Research, LLC under Contract No. 290-2014-00004-C.) AHRQ Publication No. 17(18)-EHC017-EF. Rockville, MD: Agency for Healthcare Research and Quality; June 2019. https:​//effectivehealthcare.ahrq.gov/. [PubMed: 31891455]
Lauer MS, D’Agostino RB, Sr. The randomized registry trial--the next disruptive technology in clinical research? N Engl J Med. 2013;369(17):1579–81. PMID: 23991657. DOI: 10.1056/NEJMp1310102. [PubMed: 23991657] [CrossRef]
Rao SV, Hess CN, Barham B, et al. A registry-based randomized trial comparing radial and femoral approaches in women undergoing percutaneous coronary intervention: the SAFE-PCI for Women (Study of Access Site for Enhancement of PCI for Women) trial. JACC Cardiovasc Interv. 2014;7(8):857–67. PMID: 25147030. DOI: 10.1016/j.jcin.2014.04.007. [PubMed: 25147030] [CrossRef]
U.S. Food and Drug Administration. Adaptive Designs for Clinical Trials of Drugs and Biologics. Draft Guidance for Industry. October 2018. https://www​.fda.gov/downloads​/drugs/guidances/ucm201790.pdf. Accessed June 10, 2019.
Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics. 1995;51(1):228–35. PMID: 7766778. [PubMed: 7766778]
Ryan PB, Powell GE, Pattishall EN, et al. Performance of screening multiple observational databases for active drug safety surveillance. International Society of Pharmacoepidemiology; Providence, RI, USA. 2009.
Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348–55. PMID: 27283160. DOI: 10.1093/ije/dyw098. [PMC free article: PMC5407170] [PubMed: 27283160] [CrossRef]
Norén GN, Hopstadius J, Bate A, et al. Temporal pattern discovery in longitudinal electronic patient records. Data Mining and Knowledge Discovery. 2009;20(3):361–87. DOI: 10.1007/s10618-009-0152-3. [CrossRef]
Travis LB, Rabkin CS, Brown LM, et al. Cancer survivorship--genetic susceptibility and second primary cancers: research strategies and recommendations. J Natl Cancer Inst. 2006;98(1):15–25. PMID: 16391368. DOI: 10.1093/jnci/djj001. [PubMed: 16391368] [CrossRef]
Sackett DL, Haynes RB, Tugwell P. Clinical epidemiology. Boston: Little, Brown and Company; 1985. p. 228.
Hennekens CH, Buring JE, Mayrent SL. Epidemiology in medicine. 1st ed. Boston: Little, Brown and Company; 1987.
Schoebel FC, Gradaus F, Ivens K, et al. Restenosis after elective coronary balloon angioplasty in patients with end stage renal disease: a case-control study using quantitative coronary angiography. Heart. 1997;78(4):337–42. PMID: 9404246. DOI: 10.1136/hrt.78.4.337. [PMC free article: PMC1892250] [PubMed: 9404246] [CrossRef]
Rothman K, Greenland S. Modern Epidemiology. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 1998. p. 175–9.
Oral contraceptive use and the risk of endometrial cancer. The Centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983;249(12):1600–4. PMID: 6338265. [PubMed: 6338265]
Oral contraceptive use and the risk of ovarian cancer. The Centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983;249(12):1596–9. PMID: 6338264. [PubMed: 6338264]
Long-term oral contraceptive use and the risk of breast cancer. The centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983;249(12):1591–5. PMID: 6338262. [PubMed: 6338262]
Speck CE, Kukull WA, Brenner DE, et al. History of depression as a risk factor for Alzheimer’s disease. Epidemiology. 1995;6(4):366–9. PMID: 7548342. [PubMed: 7548342]
Wacholder S, McLaughlin JK, Silverman DT, et al. Selection of controls in case-control studies. I. Principles. Am J Epidemiol. 1992;135(9):1019–28. PMID: 1595688. DOI: 10.1093/oxfordjournals.aje.a116396. [PubMed: 1595688] [CrossRef]
Wacholder S, Silverman DT, McLaughlin JK, et al. Selection of controls in case-control studies. III. Design options. Am J Epidemiol. 1992;135(9):1042–50. PMID: 1595690. DOI: 10.1093/oxfordjournals.aje.a116398. [PubMed: 1595690] [CrossRef]
Wacholder S, Silverman DT, McLaughlin JK, et al. Selection of controls in case-control studies. II. Types of controls. Am J Epidemiol. 1992;135(9):1029–41. PMID: 1595689. DOI: 10.1093/oxfordjournals.aje.a116397. [PubMed: 1595689] [CrossRef]
Rothman K, Greenland S. Modern Epidemiology. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 1998. p. 108.
Vigneswaran R, Aitchison SJ, McDonald HM, et al. Cerebral palsy and placental infection: a case-cohort study. BMC Pregnancy Childbirth. 2004;4(1):1. PMID: 15005809. DOI: 10.1186/1471-2393-4-1. [PMC free article: PMC343280] [PubMed: 15005809] [CrossRef]
Ong AT, Daemen J, van Hout BA, et al. Cost-effectiveness of the unrestricted use of sirolimus-eluting stents vs. bare metal stents at 1 and 2-year follow-up: results from the RESEARCH Registry. European heart journal. 2006;27(24):2996–3003. PMID: 17114234. DOI: 10.1093/eurheartj/ehl357. [PubMed: 17114234] [CrossRef]
Farfel M, DiGrande L, Brackbill R, et al. An overview of 9/11 experiences and respiratory and mental health conditions among World Trade Center Health Registry enrollees. J Urban Health. 2008;85(6):880–909. PMID: 18785012. DOI: 10.1007/s11524-008-9317-4. [PMC free article: PMC2587652] [PubMed: 18785012] [CrossRef]
E-Cigarette Effects on Markers of Cardiovascular and Pulmonary Disease ClinicalTrials​.gov. NCT03863509. https://clinicaltrials.gov/ct2/show/NCT03863509. Accessed June 10, 2019.
Hulley SB, Cumming SR. Designing clinical research. Baltimore: Williams & Wilkins; 1988.
Rothman KJ, Gallacher JE, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42(4):1012–4. PMID: 24062287. DOI: 10.1093/ije/dys223. [PMC free article: PMC3888189] [PubMed: 24062287] [CrossRef]
Hunter D. First, gather the data. N Engl J Med. 2006;354(4):329–31. PMID: 16436764. DOI: 10.1056/NEJMp058235. [PubMed: 16436764] [CrossRef]
Mangano DT, Tudor IC, Dietzel C, et al. The risk associated with aprotinin in cardiac surgery. N Engl J Med. 2006;354(4):353–65. PMID: 16436767. DOI: 10.1056/NEJMoa051379. [PubMed: 16436767] [CrossRef]
Ripollone JE, Huybrechts KF, Rothman KJ, et al. Implications of the Propensity Score Matching Paradox in Pharmacoepidemiology. Am J Epidemiol. 2018;187(9):1951–61. PMID: 29750409. DOI: 10.1093/aje/kwy078. [PMC free article: PMC6118075] [PubMed: 29750409] [CrossRef]
Jackson JW, Schmid I, Stuart EA. Propensity Scores in Pharmacoepidemiology: Beyond the Horizon. Curr Epidemiol Rep. 2017;4(4):271–80. PMID: 29456922. DOI: 10.1007/s40471-017-0131-y. [PMC free article: PMC5810585] [PubMed: 29456922] [CrossRef]
National Cancer Institute. Surveillance Epidemiology and End Results. https://seer​.cancer.gov/. Accessed June 10, 2019.
Becker JC, Lorenz E, Ugurel S, et al. Evaluation of real-world treatment outcomes in patients with distant metastatic Merkel cell carcinoma following second-line chemotherapy in Europe. Oncotarget. 2017;8(45):79731–41. PMID: 29108353. DOI: 10.18632/oncotarget.19218. [PMC free article: PMC5668086] [PubMed: 29108353] [CrossRef]
Metropolitan Atlanta Congenital Defects Program (MACDP). National Center on Birth Defects and Developmental Disabilities. Centers for Disease Control and Prevention. https://www​.cdc.gov/ncbddd​/birthdefects/macdp.html. Accessed June 10, 2019.
Hatfield L, Zusterzeel R, Daluwatte C, Normand SL. Improving Access To Medical Devices: The Use And Evolution Of Objective Performance Criteria. Health Affairs Blog. July 26, 2018. https://www​.healthaffairs​.org/do/10.1377/hblog20180726​.907775/full/. Accessed June 10, 2019.
U.S. Food and Drug Administration; Center for Devices and Radiological Health. Design Considerations for Pivotal Clinical Investigations for Medical Devices. Guidance for Industry, Clinical Investigators, Institutional Review Boards and Food and Drug Administration Staff. November 2013. https://www​.fda.gov/regulatory-information​/search-fda-guidance-documents​/design-considerations-pivotal-clinical-investigations-medical-devices. Accessed June 10, 2019.
Cochran WG. Sampling Techniques. 3rd ed. New York: John Wiley & Sons; 1977.
Herbst AL, Ulfelder H, Poskanzer DC. Adenocarcinoma of the vagina. Association of maternal stilbestrol therapy with tumor appearance in young women. N Engl J Med. 1971;284(15):878–81. PMID: 5549830. DOI: 10.1056/NEJM197104222841604. [PubMed: 5549830] [CrossRef]
Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990;1(1):43–6. PMID: 2081237. [PubMed: 2081237]
Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–92. PMID: 10861325. DOI: 10.1056/NEJM200006223422507. [PMC free article: PMC1557642] [PubMed: 10861325] [CrossRef]
Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86. PMID: 10861324. DOI: 10.1056/NEJM200006223422506. [PubMed: 10861324] [CrossRef]
Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312(7040):1215–8. PMID: 8634569. DOI: 10.1136/bmj.312.7040.1215. [PMC free article: PMC2350940] [PubMed: 8634569] [CrossRef]
Rothman K. Modern Epidemiology. Boston: Little Brown and Company; 1986. p. 83.
Kho ME, Duffett M, Willison DJ, et al. Written informed consent and selection bias in observational studies using medical records: systematic review. BMJ. 2009;338:b866. PMID: 19282440. DOI: 10.1136/bmj.b866. [PMC free article: PMC2769263] [PubMed: 19282440] [CrossRef]
Petri H, Urquhart J. Channeling bias in the interpretation of drug effects. Stat Med. 1991;10(4):577–81. PMID: 2057656. [PubMed: 2057656]
Johnston SC. Identifying confounding by indication through blinded prospective review. Am J Epidemiol. 2001;154(3):276–84. PMID: 11479193. DOI: 10.1093/aje/154.3.276. [PubMed: 11479193] [CrossRef]
Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98(3):253–9. PMID: 16611199. DOI: 10.1111/j.1742-7843.2006.pto_293.x. [PMC free article: PMC1790968] [PubMed: 16611199] [CrossRef]
Sturmer T, Joshi M, Glynn RJ, et al. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol. 2006;59(5):437–47. PMID: 16632131. DOI: 10.1016/j.jclinepi.2005.07.004. [PMC free article: PMC1448214] [PubMed: 16632131] [CrossRef]
Loughlin J, Seeger JD, Eng PM, et al. Risk of hyperkalemia in women taking ethinylestradiol/drospirenone and other oral contraceptives. Contraception. 2008;78(5):377–83. PMID: 18929734. DOI: 10.1016/j.contraception.2008.06.012. [PubMed: 18929734] [CrossRef]
Brookhart MA, Rassen JA, Schneeweiss S. Instrumental variable methods in comparative safety and effectiveness research. Pharmacoepidemiol Drug Saf. 2010;19(6):537–54. PMID: 20354968. DOI: 10.1002/pds.1908. [PMC free article: PMC2886161] [PubMed: 20354968] [CrossRef]
Evans W, Ringel J. Can higher cigarette taxes improve birth outcomes? J Public Economics. 1997;72(1):135–54. DOI: 10.3386/w5998. [CrossRef]
Schneeweiss S, Seeger JD, Landon J, et al. Aprotinin during coronary-artery bypass grafting and risk of death. N Engl J Med. 2008;358(8):771–83. PMID: 18287600. DOI: 10.1056/NEJMoa0707571. [PubMed: 18287600] [CrossRef]
Secemsky EA, Kirtane A, Bangalore S, et al. Use and Effectiveness of Bivalirudin Versus Unfractionated Heparin for Percutaneous Coronary Intervention Among Patients With ST-Segment Elevation Myocardial Infarction in the United States. JACC Cardiovasc Interv. 2016;9(23):2376–86. PMID: 27838271. DOI: 10.1016/j.jcin.2016.09.020. [PubMed: 27838271] [CrossRef]
Vertullo CJ, Graves SE, Peng Y, et al. The effect of surgeon’s preference for hybrid or cemented fixation on the long-term survivorship of total knee replacement. Acta Orthop. 2018;89(3):329–35. PMID: 29528754. DOI: 10.1080/17453674.2018.1449466. [PMC free article: PMC6055785] [PubMed: 29528754] [CrossRef]
Bosco JL, Silliman RA, Thwin SS, et al. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. J Clin Epidemiol. 2010;63(1):64–74. PMID: 19457638. DOI: 10.1016/j.jclinepi.2009.03.001. [PMC free article: PMC2789188] [PubMed: 19457638] [CrossRef]
Dreyer NA, Velentgas P, Westrich K, et al. The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution. J Manag Care Spec Pharm. 2014;20(3):301–8. PMID: 24564810. DOI: 10.18553/jmcp.2014.20.3.301. [PMC free article: PMC10437555] [PubMed: 24564810] [CrossRef]
Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol. 2003;158(9):915–20. PMID: 14585769. DOI: 10.1093/aje/kwg231. [PubMed: 14585769] [CrossRef]
Kristman V, Manno M, Cote P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol. 2004;19(8):751–60. PMID: 15469032. [PubMed: 15469032]
Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data: Springer; 2009.
U.S. Food and Drug Administration. Unique Device Identification System Final Rule. September 24, 2013. https://www​.federalregister​.gov/documents​/2013/09/24/2013-23059​/unique-device-identification-system. Accessed June 10, 2019.
Gliklich RE, Dreyer NA, Leavy MB, Christian JB (eds). 21st Century Patient Registries. EBook addendum to Registries for Evaluating Patient Outcomes: A User’s Guide, 3rd Edition. (Prepared by L&M Policy Research, LLC under Contract No. 290-2014-00004-C.) AHRQ Publication No. 17(18)-EHC013-EF. Rockville, MD: Agency for Healthcare Research and Quality; February 2018. www​.effectivehealthcare.ahrq.gov. [PubMed: 29708678]
Dreyer NA, Blackburn SC, Mt-Isa S, et al. Direct-to-Patient Research: Piloting a New Approach to Understanding Drug Safety During Pregnancy. JMIR Public Health Surveill. 2015;1(2):e22. PMID: 27227140. DOI: 10.2196/publichealth.4939. [PMC free article: PMC4869223] [PubMed: 27227140] [CrossRef]
Cragan JD, Gilboa SM. Including prenatal diagnoses in birth defects monitoring: Experience of the Metropolitan Atlanta Congenital Defects Program. Birth Defects Res A Clin Mol Teratol. 2009;85(1):20–9. PMID: 19089857. DOI: 10.1002/bdra.20508. [PubMed: 19089857] [CrossRef]
Lutgens M, van Oijen M, Mooiweer E, et al. A risk-profiling approach for surveillance of inflammatory bowel disease-colorectal carcinoma is more cost-effective: a comparative cost-effectiveness analysis between international guidelines. Gastrointest Endosc. 2014;80(5):842–8. PMID: 25088918. DOI: 10.1016/j.gie.2014.02.1031. [PubMed: 25088918] [CrossRef]

Case Examples for Chapter 3

Case Example 5. Using a Registry To Assess Long-Term Product Safety

DescriptionThe British Society for Rheumatology Biologics Registers in Rheumatoid Arthritis (BSRBR-RA) is a prospective observational study conducted to monitor the routine clinical use and long-term safety of biologics (including biosimilars) and other targeted therapies in patients with rheumatoid arthritis and other rheumatic conditions.

Research Governance Sponsor: The University of Manchester

Funder: The British Society for Rheumatology (BSR)

Year Started2001
Year EndedOngoing
No. of SitesAll consultant rheumatologists in the UK who have prescribed anti-TNF and other targeted therapies have an opportunity to participate in the register.
No. of PatientsMore than 30,000


Rheumatoid arthritis (RA) is a progressive inflammatory disease characterized by joint damage, pain, and disability. Among the pharmacologic treatments, nonbiologic disease-modifying antirheumatic drugs (DMARDs) are considered the first-line treatment. Biologic therapies were introduced approximately 20 years ago and offered patients and providers a new class of agents with demonstrated efficacy in RA patients. The most commonly used biologics are tumor necrosis factors (TNF) inhibitors (etanercept, infliximab, adalimumab, certolizumab, and golimumab), although the use of other classes of advanced therapies, including andi-CD20 (rituximab), anti-IL6 (tocilizumab, sarilumab), and JAK inhibitors (baricitinib, tofacitinib) is also increasing. However, results from clinical trials and pharmacovigilance studies raised potential safety concerns, and limited long-term data on these therapies are available at the time of regulatory approval. Of particular concern is the risk of serious infections including tuberculosis and malignancy.

Proposed Solution

A prospective observational registry was launched in 2001 to monitor the safety and effectiveness of biologic treatments. This United Kingdom-wide national project was launched after the introduction of the first tumor necrosis factors (TNF) alpha inhibitors and has now expanded to include therapies across a wide range of biologic and targeted therapies, including the recent inclusion of biosimilar drugs. The registry collects data on response to treatment and adverse events (AEs) every six months, and patients are followed for the life of the registry. In addition to patients receiving biologic and targeted therapies, the registry has enrolled a control cohort of patients receiving nonbiologic DMARDs, although recruitment to this cohort (n=3800) ended in 2008.


The registry has now published over 60 papers looking at a wide range of outcomes, including treatment effectiveness and safety, such as the risk of infections, malignancy, cardiovascular disease, and thromboembolic disease. Details of publications, study protocols and further information can be found on the registry website. Datasets can be made available to researchers upon approval of an application, and information on how to apply for a dataset may found on the registry website.

Key Point

As novel drugs and treatments are developed and licensed, registries may be useful tools for collecting long-term data to assess known and emerging safety concerns.

For More Information

  • Nikiphorou E, Buch MH, Hyrich KL. Biologics registers in RA: methodological aspects, current role and future applications. Nat Rev Rheumatol. 2017 Aug;13(8):503–510. doi: 10.1038/nrrheum.2017.81. PMID: 28569267. DOI: 10.1038/nrrheum.2017.81. [PubMed: 28569267] [CrossRef] [CrossRef]

Case Example 6. Expanding an Ongoing Pregnancy Registry

DescriptionThe Antiretroviral Pregnancy Registry is the oldest ongoing pregnancy exposure registry. This multisponsor, international, voluntary, collaborative registry monitors prenatal exposures to all marketed antiretroviral drugs for potential risk of birth defects.
SponsorsAbbVie, Accord Healthcare Inc., Alvogen Inc., Amneal Pharmaceuticals LLC, Apotex Inc., Aurobindo Pharma Ltd., Boehringer Ingelheim Pharmaceuticals Inc., Bristol-Myers Squibb Company, Celltrion, Inc., Cipla Ltd., F. Hoffman La-Roche, Gilead Sciences Inc., Hetero Labs Ltd., Hikma Pharmaceuticals USA Inc., Janssen Scientific Affairs, Lannett Company Inc., Lupin Pharmaceuticals, Macleods Pharmaceuticals Ltd., Merck & Co. Inc., Mylan Laboratories, Novartis Pharmaceuticals, Prinston, Qilu Pharmaceutical Co, Ltd., Sandoz Inc, SigmaPharm Laboratories LLC, Strides Shasun Ltd., Teva Pharmaceuticals USA Inc., and ViiV Healthcare
Year Started1989
Year EndedOngoing
No. of SitesNot site-based; open to all healthcare providers.
No. of Patients20,375


Antiretroviral treatments represent an area of particular concern for monitoring safety in pregnancy. Women may need to take the drugs during pregnancy to manage their own HIV infection and to reduce the risk of transmitting HIV to the infant, but these benefits must be weighed against the risk of teratogenic effects. Because of these factors, it is extremely important for clinicians and patients to understand the risks of using antiretroviral drugs during pregnancy in order to make an informed decision. However, ethical and practical concerns make a randomized trial to gather these data difficult, if not impossible.

In 1989, the first manufacturer of an antiretroviral drug voluntarily initiated a pregnancy exposure registry to track the outcomes of women who had used its product during pregnancy. The purpose of the registry is to collect information on any teratogenic effects of the product by prospectively enrolling women during the course of their pregnancy and following up with them to determine the outcome of the pregnancy. Physicians enroll a patient by providing information on the pregnancy dates, characteristics of the HIV infection, drug dosage, length of therapy, and trimester of exposure to the antiretroviral drug. Information on the pregnancy outcome is gathered through a followup form sent to the physician after the expected delivery date.

In 1993, the registry was expanded to include all antiretroviral drugs, as other manufacturers voluntarily joined the registry once their drugs were on the market. The registry is international in scope and allows any healthcare provider to enroll a patient who intentionally or unintentionally has used an antiretroviral drug during pregnancy. The U.S. Food and Drug Administration, which has used this registry as a model for new pregnancy registries, now requires all new and generic antiretroviral drugs marketed in the U.S to be monitored in a registry.

The year 2019 marks 30 years of active enrollment with the registry expanding to now monitor 153 antiretroviral drugs including 55 brand-name single-entity or fixed-dose combinations and 98 generic versions from 28 companies. The registry has also increased enrollment as well as its geographic representation by incorporating the datasets of comparable, completed epidemiological studies. For example, the registry added data on nearly 1,000 women from a study conducted in Brazil and Argentina of antiretroviral-exposed pregnant women who delivered between the years 2002 and 2007. In addition, electronic data capture (EDC) was introduced in 2010 as a data collection method for the registry.

In summary, early challenges for the registry included establishing standard processes for monitoring and assessing the safety of drugs during pregnancy. Key challenges in recent years have included managing the methodological and analytic implications of a rapid growth in size, complex drug regimens and the operational implications of long-term EDC system management.

Proposed Solution

To ensure both rigor and consistency early on, the registry put in place predefined analytic methods and criteria for recognizing a potential teratogenic signal. Tools for coding and classifying birth defects were developed for the registry to maximize the likelihood of identifying a teratogenic signal. This unique system groups birth defects by etiology or embryology rather than by general location or category, as does the Medical Dictionary for Regulatory Activities (MedDRA). Grouping like defects together increases the likelihood of detecting a potential signal. The registry also codes the temporal association between timing of exposure and formation of the birth defect, aiding in signal detection.

Specific monitoring criteria were developed for evaluating signals at various levels, including the Rule of Three (the rule that three exposure-specific cases with the same birth defect require immediate evaluation). This rule is based on the statistical principle that the likelihood of finding at least three of any specific defect in a cohort of 600 or fewer by chance alone is less than 5 percent.

More recently, large increases in enrollment required re-evaluation of the adequacy of existing signal detection rules. The Rule of Three continues to serve an important role; however, understanding weak signals is methodologically challenging. Incorporating enrollments from comparable epidemiological studies into the registry population has boosted enrollment, increased cultural diversity, and enhanced signal detection capabilities. Each merger of external data prompts the need to re-examine the potential for selection and ascertainment bias.

Operationally, each new participating manufacturer undergoes a series of trainings and is required to obtain institutional review board approval before participation in the registry. Registry trainings and standard operating procedures are reviewed at biannual steering committee meetings and revised as appropriate.

In expanding the options for data entry into the registry, a hybrid EDC-paper approach was deemed operationally feasible in lieu of an EDC-only approach. This allowed a subset of established reporters to use EDC, while limiting disruption for reporters who preferred to report data on paper CRFs.


The registry now contains data on 20,375 prospective pregnancies with exposure to 55 medications. Registry data have been used in 15 publications, 15 presentations, and more than 40 conference abstracts and posters, and the registry design and operation have been the subject of many publications and presentations. The registry findings can help provide clinicians and patients with information to make informed decisions regarding use of antiretroviral drugs during pregnancy.

Key Point

A pregnancy exposure registry can employ continuous quality improvement practices to identify and define key quality processes and keep the registry current and innovative throughout its life cycle. The fact that the registry had established, standard policies and procedures for coding, monitoring, and analysis was critical in incorporating new partners and data sources quickly and easily. Regular review of these policies and procedures is essential to respond to the changing registry environment.

For More Information

  • Antiretroviral Pregnancy Registry Steering Committee. Antiretroviral Pregnancy Registry International Interim Report for 1 January 1989 through 31 January 2019. Wilmington, NC: Registry Coordinating Center; 2019. http://www.apregistry.com/. Accessed June 10, 2019.
  • Tilson H, Roberts S, Watts H, et al. The Antiretroviral Pregnancy Registry: A 20th anniversary celebration. Pharmacoepidemiology and Drug Safety. 2011;20(S1):S190. PMID: 17229330. DOI: 10.1097/01.ogx.0000253377.14647.80. [CrossRef]
  • Tilson H, Doi PA, Covington DL, et al. The antiretrovirals in pregnancy registry: A fifteenth anniversary celebration. Obstet Gynecol Surv. 2007;62:137–48. PMID: 17229330. DOI: 10.1097/01.ogx.0000253377.14647.80. [PubMed: 17229330] [CrossRef]
  • Covington D, Tilson H, Elder J, et al. Assessing teratogenicity of antiretroviral drugs: monitoring and analysis plan of the Antiretroviral Pregnancy Registry. Pharmacoepidemiol Drug Saf. 2004;13:537–45. PMID: 15317035. DOI: 10.1002/pds.982. [PubMed: 15317035] [CrossRef]
  • Scheuerle A, Covington D. Clinical review procedures for the Antiretroviral Pregnancy Registry. Pharmacoepidemiol Drug Saf. 2004;13:529–36. PMID: 15317034 DOI: 10.1002/pds.971. [PubMed: 15317034] [CrossRef]

Case Example 7. Designing a Registry To Address Unique Patient Enrollment Challenges

DescriptionThe Anesthesia Awareness Registry is a survey-based registry that collects detailed data about patient experiences of anesthesia awareness. Patient medical records are used to assess anesthetic factors associated with the patient’s experience. An optional set of psychological assessment instruments measure potential trauma-related sequelae including depression and post-traumatic stress disorder (PTSD).
SponsorAmerican Society of Anesthesiologists
Year Started2007
Year EndedOngoing
No. of SitesNot applicable
No. of Patients366


Anesthesia awareness is a recognized complication of general anesthesia, defined as the unintended experience and explicit recall of events during surgery. The incidence of anesthesia awareness has been estimated at 1–2 patients per 1,000 anesthetics and may result in development of serious and long-term psychological sequelae including PTSD. The causes of the phenomenon and preventive strategies have been studied, but there is disagreement in the scientific community about the effectiveness of monitoring devices for prevention of anesthesia awareness.

The population of patients experiencing anesthesia awareness is difficult to identify. Although standard short questionnaires designed to identify anesthesia awareness are sometimes administered to patients postoperatively, many patients experience delayed recollection and do not realize that they were awake during their procedure until several weeks later. These patients may or may not report their experience to their provider. In addition, because of the often unsettling and traumatic nature of their experience, even patients who recognize their anesthesia awareness before being discharged from the hospital may not feel comfortable reporting it to their surgeon or other healthcare providers.

With ongoing coverage in the media, anesthesiologists were facing increasing concern and fear about anesthesia awareness among their patients. The American Society of Anesthesiologists sought a patient-oriented approach to this problem.

Proposed Solution

Because this population of patients is not always immediately recognized in the healthcare setting, the registry was created to collect case reports of anesthesia awareness directly from patients. A patient advocate was invited to consult in the registry’s development and provides ongoing advice from the patient perspective. The registry hosts a website that provides information about anesthesia awareness and directions for enrolling in the registry. Any patient who believes they have experienced anesthesia awareness may voluntarily submit a survey and medical records to the registry. Psychological assessments are optional. An optional open-ended discussion about the patient’s anesthesia awareness experience provides patients with an opportunity to share information that may not be elicited through the survey.


The registry has enrolled 366 patients since 2007. Patients who enroll are self-selected, and the sample is likely biased towards patients with emotional sequelae. While the information provided to potential enrollees clearly states that eligibility is restricted to awareness during general anesthesia, a surprising number of enrollments are patients who were supposed to be awake during regional anesthesia or sedation. This revealed a different side to the problem of anesthesia awareness: clearly, some patients did not understand the nature of the anesthetic that would be provided for their procedure, or patients had expectations that were not met by their anesthesia providers. Most enrollees experienced long-term psychological sequelae regardless of anesthetic technique.

Key Point

Allowing the registry’s purpose to drive its design produces a registry that is responsive to the expected patient population. Employing direct-to-patient recruitment can be an effective way of reaching a patient population that otherwise would not be enrolled in the registry, and can yield surprising and important insights into patient experience.

For More Information

  • Domino KB. Committee on Professional Liability opens anesthesia awareness registry. ASA Newsletter. 2007. p. 29.p. 34.
  • Kent CD, Metzger NA, Posner KL, et al. Anesthesia Awareness Registry: psychological impacts for patients. Anesthesiology. 2011:A003.
  • Domino KB, Metzger NA, Mashour GA. Anesthesia Awareness Registry: patient responses to awareness. Br J Anaesth. 2012;108(2):338P.
  • Kent CD, Mashour GA, Metzger NA, et al. Psychological impact of unexpected explicit recall of events occurring during surgery performed under sedation, regional anaesthesia, and general anaesthesia: data from the Anesthesia Awareness Registry. Br J Anaesth. 2013;110(3):381–7. PMID: 23161356. DOI: 10.1093/bja/aes386. [PubMed: 23161356] [CrossRef]
  • Kent CD, Posner KL, Mashour GA, et al. Patient perspectives on intraoperative awareness with explicit recall: report from a North American anaesthesia awareness registry. Br J Anaesth. 2015;115 Suppl 1:i114–i21. PMID: 26174296. DOI: 10.1093/bja/aev211. [PubMed: 26174296] [CrossRef]

Case Example 8. Using Registries To Drive Quality Improvement in Chronic Conditions

DescriptionThe National Parkinson Foundation Quality Improvement Initiative is a registry-based quality care program that captures longitudinal data on clinical interventions and patient-reported outcomes to identify, implement, and disseminate best practices for the treatment and management of Parkinson’s disease.
SponsorNational Parkinson Foundation
Year Started2009
Year EndedOngoing
No. of Sites21 sites in North America, the Netherlands, and Israel
No. of Patients>8,000 patients


Parkinson’s disease (PD), an incurable, progressive neurogenerative disorder associated with a high burden of disease, presents unique challenges for quality improvement initiatives. Treatments for PD generally focus on reducing patients’ symptoms and improving quality of life. Unlike other chronic conditions where improvement can be measured in terms of well-defined outcomes such as survival or cardiovascular events, quality improvement in PD can best be measured using patient-based outcomes. However, identifying appropriate patient-based outcomes for this disease can be a challenge. In addition, variability exists in the clinical diagnosis, management, and treatment of PD. Studies have shown that PD patients treated by a neurologist experience better outcomes, such as a decrease in hip fractures or nursing home placement. However, the specific management and treatment strategies used by these specialists have not been studied or well-described. The lack of evidence-based treatment standards warranted a data-driven approach to identify and understand best practices that improve the quality of care and quality of life for PD patients.

Proposed Solution

In 2009, the National Parkinson Foundation launched an initiative to improve the quality of care in PD. To support an evidence-based approach, the foundation initiated a PD registry to capture clinical interventions and patient-reported outcomes over time from multiple centers across the United States, Canada, and internationally. The initiative, led by a steering committee of movement disorders neurologists, is a unique effort in PD research because of its ability to collect long-term, longitudinal data from multiple centers and its focus on patient-based outcomes data, rather than process of care measures. The aims of the registry are to accelerate clinical discovery, promote collaborative science, and drive advancements in clinical practice toward patient-centered care.


The registry includes data on more than 8,000 patients from 21 centers. Patients’ encounter-based data, including demographics, comorbidities, hospitalizations, falls, medications, treatments, and outcomes, are collected annually on brief data collection forms. The registry database includes a diverse population of PD patients, and analyses have confirmed variation in practice patterns across centers. The registry data have yielded important findings, including enhanced understanding of factors and predictors of patients’ quality of life and caregiver burden. Additional cross-sectional and longitudinal analyses are planned using physician care and patient outcome data to describe practice patterns across the registry, identify and improve understanding of best practices, and support the development of guidelines.

Many neurologists were initially doubtful about the value of a registry in this disease area. For the most part, their past experience was with mortality-based registries based around interventions or fatal illnesses; these failed to model a disease with complex, heterogeneous symptomology, where the pathology could not be directly measured. Increasingly providers have recognized the value of the statistical power and nuanced insight that can be leveraged in this large and detailed registry of expert care.

Key Point

Registry-based quality improvement programs can be useful in many clinical settings, from in-hospital care (e.g., heart failure) to chronic progressive diseases (e.g., PD). The design of the registry and the quality improvement initiative must reflect the nature of the disease and the state of existing evidence. For chronic, progressive diseases, registries can be useful tools for identifying, developing, and disseminating guidelines for best practices to improve quality of care.

For More Information

  • Margolius A, Cubillos F, He Y, et al. Predictors of clinically meaningful change in PDQ-39 in Parkinson’s disease. Parkinsonism & related disorders. 2018;56:93–7. PMID: 30056039 DOI: 10.1016/j.parkreldis.2018.06.034. [PubMed: 30056039] [CrossRef]
  • Rafferty MR, Schmidt PN, Luo ST, et al. Regular Exercise, Quality of Life, and Mobility in Parkinson’s Disease: A Longitudinal Analysis of National Parkinson Foundation Quality Improvement Initiative Data. J Parkinsons Dis. 2017;7(1):193–202. PMID: 27858719. DOI: 10.3233/JPD-160912. [PMC free article: PMC5482526] [PubMed: 27858719] [CrossRef]
  • Hassan A, Wu SS, Schmidt P, et al. The Profile of Long-term Parkinson’s Disease Survivors with 20 Years of Disease Duration and Beyond. Journal of Parkinson’s disease. 2015;5(2):313–9. PMID: 25720446. DOI: 10.3233/JPD-140515. [PubMed: 25720446] [CrossRef]
  • Parashos SA, Wielinski CL, Giladi N, et al. Falls in Parkinson disease: analysis of a large cross-sectional cohort. Journal of Parkinson’s disease. 2013;3(4):515–22. PMID: 24113557 DOI: 10.3233/JPD-130249. [PubMed: 24113557] [CrossRef]
©2020 United States Government, as represented by the Secretary of the Department of Health and Human Services, by assignment.

All rights reserved. The Agency for Healthcare Research and Quality (AHRQ) permits members of the public to reproduce, redistribute, publicly display, and incorporate this work into other materials provided that it must be reproduced without any changes to the work or portions thereof, except as permitted as fair use under the U.S. Copyright Act. This work contains certain tables and figures noted herein that are subject to copyright by third parties. These tables and figures may not be reproduced, redistributed, or incorporated into other materials independent of this work without permission of the third-party copyright owner(s). This work may not be reproduced, reprinted, or redistributed for a fee, nor may the work be sold for profit or incorporated into a profit-making venture without the express written consent of AHRQ. This work is subject to the restrictions of Section 1140 of the Social Security Act, 42 U.S.C. § 1320b-10. When parts of this work are used or quoted, the following citation should be used:

Bookshelf ID: NBK562568


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (4.0M)

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...