NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Roundtable on Research and Development of Drugs, Biologics, and Medical Devices; Davis JR, Nolan VP, Woodcock J, et al., editors. Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making: Workshop Report. Washington (DC): National Academies Press (US); 1999.

Cover of Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making

Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making: Workshop Report.

Show details

Final Comments

In 1997, 49 new therapeutic compounds for 39 diseases affecting an estimated 160 million people were introduced onto the market. Drug development is a lengthy and uncertain process, taking up to 15 years, with less than 1 in 10,000 compounds making it from preclinical testing to marketing. This process is also expensive, costing at least $300 million and possibly more than $500 million for each drug that successfully makes it to market. An individual clinical trial can cost as much as $100 million. Because at least 15 percent of clinical trial expenses are related to monitoring activities, however, more prudent and efficient monitoring has the potential to substantially reduce the costs of clinical trials.

An important way to improve monitoring is to reduce the amount of data collected. An average 12-month clinical trial with 2,000 patients will generate up to 3 million data points. A potential 10 million opportunities for error are estimated per trial, given that the data are handled at least six times, for example, in the clinic, during double data entry, and during cleanup activities. Even with a "good" error rate of 1 in 1,000, this would yield 10,000 errors. However, actual error rates are often much higher. Therefore, sponsors should be collecting only those data that are directly related to the outcome variable. Collection of superfluous data generates an enormous number of errors and may compromise a sponsor's and the Food and Drug Administration's (FDA's) ability to interpret the results accurately.

The challenge at hand is to collect the correct data and to monitor the data collection process more effectively and efficiently. Key elements in this effort include the following:

  • Engineer data quality into the process by creating systems that limit the opportunity for errors.
  • Standardize formats and procedures, where possible, to increase efficiency.
  • Simplify the experimental design because complex studies have the potential to yield more errors.
  • Plan ahead by defining the proper data set needed and specifying requirements for data quality (e.g., error rates of 1 to 5 per 1,000 for primary end-points, but 2 or even 5 errors per 100 for secondary endpoints).
  • Clarify expectations by discussing with regulators the types and amount of data collected, the extent of monitoring, and the methods for data analysis.

A major theme that emerged from the workshop was that of partnerships. It was proposed that industry sponsors work with FDA before a clinical trial to define a coherent set of data that will demonstrate safety and efficacy and to set up an appropriate monitoring plan to ensure the quality of those data. Although FDA has been open to such efforts and considerable progress has been made in several areas, there is room for continued improvement. Other points emerged during the workshop and are described below.


During the past 20 years the relationship between FDA and the pharmaceutical industry has evolved from an adversarial one to a more collaborative interaction. This strengthened relationship now provides an opportunity for FDA and industry to move forward on the issues of hierarchy in data quality assessment, early planning to build quality into the process, and improved communication.

Education and Training

Education of the public on the FDA review process and on the technical obstacles that industry must face when it develops even a single drug and brings it to market is important for enlisting consumer confidence. However, an even greater need is education of investigators, Institutional Review Boards, industry sponsors, and even regulators. They need to be educated that the goal is not merely passage of an inspection, but rather the generation of quality data that will support the inferences drawn from a study. There may also be a need for more specific training (and possibly credentialing) of clinical investigators, as well as clinical research associates and clinical research coordinators.


The pharmaceutical industry spends substantial amounts of money on monitoring activities not required by FDA. Consequently, industry needs to be more forthcoming with FDA about its concerns and expectations. Conversely, FDA needs to reach internal consensus about its needs and expectations and then communicate that consensus externally. Similarly, FDA, sponsors, and investigators must communicate earlier and far more candidly about their expectations with regard to (1) the kinds of data that are most important to the outcome of the study, (2) adequate or acceptable ways to generate those data, and (3) the standards against which those data will be evaluated. There is also a need for more assessment and candid communication on how reimbursement issues affect data quality.


Definition of permissible error rates and the level or degree of data quality was a central theme throughout the workshop. Unfortunately, little information in the public domain defines data quality. Moreover, definitions of error rates and quality data may vary among the training, monitoring, and auditing phases of an investigation. Consensus on definitions needs to be developed for each phase of the clinical trial.

Functional Quality

The goal of data quality is not efficiency, but is reliability to provide adequate information to support the inferences made in the application. Perhaps the application should include a new section describing the steps taken in a clinical trial to achieve a high level of data quality. The audit function may be an iterative mechanism for the identification of better processes and better outcomes.

International Data

International clinical trials are becoming increasingly important sources of data for drug development and marketing. More attention needs to be devoted to the monitoring and review of internationally derived data, as well as therapeutic efficacy differences.

Other Areas for Consideration

The workshop provided an overview of the collection, validation, monitoring, and FDA review of the clinical trial data necessary for the development of a new drug product. It was successful in assessing the current situation and identifying potential solutions to many of the problems outlined in the workshop discussions. For instance, it was established that the situation would improve greatly and costs would diminish if the amount of data collected were diminished and if the data to be analyzed were prioritized in a hierarchy of importance and relevance over the long term. Additionally, the workshop outlined some of the challenges with regard to the quality and validity of data from clinical trials that lie ahead. These points, some of which are addressed below, will warrant further consideration.

It became evident during the discussions not only that it is important to plan for clinical trials, but also that it is critical that data quality be addressed in terms of postmarketing surveillance for drug safety and effectiveness for the development of drugs in general. Such surveillance is becoming increasingly important because it monitors drug use under non-ideal, real-life conditions and is significant in making Phase 3 clinical trials more meaningful in terms of determining the number and length of trials. Multiple instances of drug recall or modifications of drug labeling on the basis of observations made during postmarketing drug surveillance in the United States warrant the need for this type of activity to prevent adverse drug reactions.*

The quality of the clinical investigators was thoroughly discussed at the workshop, at which participants pointed out that many are inexperienced and not formally trained to perform clinical trials. A related challenge concerns the quality of the FDA reviewers of clinical data. There is considerable variability in experience, training, and expertise among FDA reviewers. Although FDA reviewers are generally skilled in determining how best to design a protocol, this may not always be the case due to the complexity of the process. FDA biostatisticians, who may not always be open to newer methods of data analysis, may also encounter difficulties in providing the best protocol. These are important issues that may warrant further investigation.

Another issue that bears consideration as an outgrowth of the workshop concerns the accrual of patients in clinical trials, which is a significant issue in the rate of development of a new medication. Accrual may not be an issue in terms of the quality of clinical data. However, if investigators are certified to conduct clinical trials, it is essential that there be a sufficient number in each medical discipline throughout various geographic locations to provide an adequate number of patients in a reasonable period of time. A challenging situation results when physicians are not willing to transfer eligible patients to a clinical investigator for participation in a trial because of lost income. A solution could include the conduct of clinical trials in managed care facilities.

As alluded to in some of the workshop discussions, the issue of site initiation visits is an important consideration in terms of data quality. Many site initiation visits are made by contract research organizations, and in general such organizations may be better equipped at instructing the clinical coordinators than are the clinical investigators. The quality of data may therefore improve if medical monitors perform the site initiation visits.

Another issue that emerged from the discussions concerned blinded studies. Although the extent of an FDA reviewer's data audit is influenced by numerous factors, such as blinding of study designs, objectivity of the endpoints, and whether the trial is designed to demonstrate equivalence or superiority, lack of blinding increases concern about many aspects of the treatment and assessment of the patients in a study. This implies that an in-house safety evaluation is not possible with blinded studies. One remedial action could be establishment of an external safety monitoring committee in the protocol.

Error rates were heavily discussed at the workshop. FDA, for instance, is fully aware that errors will likely occur in the clinical trial process. FDA is also aware that the occurrence of errors does not indicate that fraud has played a part in the trial. A reasonable number of minor errors is acceptable, as long as the errors do not compromise the reliability of the overall data set or the inferences that are being drawn from the data about the safety and effectiveness of the product. Although an error rate of 1 in 1,000 is considered good, actual error rates are often much higher. Because the collection of excessive data generates a large number of errors, compromising a sponsor's and FDA's ability to interpret the results accurately, it was suggested that only data directly related to the outcome variable be collected. A related issue that is worthy of consideration is the fact that the greatest errors generally occur when a notation describing a patient's incident is incorporated into the medical record. Additionally, the medical record frequently includes contradictory information because it represents the subjective account of an individual provider in an unstructured format.

Participants indicated that sponsors frequently fail to report problems or take corrective action with respect to monitoring of clinical sites and noncompliant clinical investigators, largely in part because regulations are vague and the appropriate sanctions are not defined. In many cases, it has been found that sponsors exclude the data but do not terminate trial activities at the site. A point to be considered in such cases is the fact that exclusion of data creates problems. With patients entered into the randomization scheme, removal of patients after randomization jeopardizes the basis for statistical inference. Additionally, exclusion of patients after randomization could jeopardize the operant's inference, which would be comparable to the inclusion of imperfect data.

The revised compliance guidance manual used by FDA suggests that investigators maintain copies of all source data and documents. Given the nature of medical practices in the United States, it is not uncommon for practices to be bought, sold, or go out of business within a matter of months. An issue worthy of consideration is identification of who owns the data and documents when medical practices go through these changes. It is necessary to propose some type of plan to deal effectively with such occurrences.



For more information, refer to Lazarou J, B Pomeranz, and P Corey. Incidence of adverse drug reactions in hospitalized patients. JAMA 279(15):1200–1205, 1998, and Wood A, CM Stein, and R Woosley. Making medicines safer—the need for an independent drug safety board. The New England Journal of Medicine 339(25):1851–1854, 1998.

Copyright 1999 by the National Academy of Sciences. All rights reserved.
Bookshelf ID: NBK224576


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (396K)
  • Disable Glossary Links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...