• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2011 Mar-Apr; 18(2): 173–180.
Published online Jan 26, 2011. doi:  10.1136/jamia.2010.010306
PMCID: PMC3116250

The case for randomized controlled trials to assess the impact of clinical information systems

Abstract

There is a persistent view of a significant minority in the medical informatics community that the randomized controlled trial (RCT) has a limited role to play in evaluating clinical information systems. A common reason voiced by skeptics is that these systems are fundamentally different from drug interventions, so the RCT is irrelevant. There is an urgent need to promote the use of RCTs, given the shift to evidence-based policy and the need to demonstrate cost-effectiveness of these systems. The authors suggest returning to first principles and argue that what is required is clarity about how to match methods to evaluation questions. The authors address common concerns about RCTs, and the extent to which they are fallacious, and also discuss the challenges of conducting RCTs in informatics and alternative study designs when randomized trials are infeasible. While neither a perfect nor universal evaluation method, RCTs form an important part of an evaluator's toolkit.

Background

Clinical information systems have been designed for many purposes, including decision support, patient monitoring, order communication, electronic prescribing and information retrieval. They are powerful and expensive, so their impact should be measured, not just assumed.1 Many developers and evaluators of clinical information systems choose not to use randomized controlled trials (RCT) to assess their impact. Despite JAMIA's ‘plea for controlled trials’ in 1994,2 a persistent view remains among a significant minority of medical informatics researchers that the RCT has a limited role in evaluations of information technology.3–9

Some textbooks argue that the RCT is irrelevant in medical informatics6–8 and provide examples of their ‘perils and pitfalls.’8 A common reason given for this is that ‘clinical information systems are a different kind of intervention from drugs, and techniques used to evaluate drugs (particularly RCTs) are not always appropriate.’4 We believe there is an urgent need to promote the use of RCTs of clinical information systems, given continuing reservations in the medical informatics community.

Some have called for a fundamental change or a more complex strategy in the way medical informatics applications are evaluated.5 8 10–13 However, there is a need to first define clearly the types of questions that need to be solved and then design an appropriate study to solve them. The study design can be chosen or adapted from a large and diverse set of existing quantitative and qualitative methods of evaluation.14–17 It is a waste of energy, money and time to reinvent the wheel.

Surprisingly, the uncontrolled before-and-after study is still commonly used to assess the impact of clinical information systems.18–21 In Copi and Cohen's classic text on logic, now in its 14th edn and almost 60 years old, the authors state that ‘any reasoning that relies on treating as the cause of a thing what is not really its cause must be seriously mistaken,’ a mistake they referred to as ‘false cause.’22 They then go on to say that ‘the most common variety of false cause is the error of concluding that an event is caused by another simply because it follows the other … This variety of false cause is widely called the fallacy of post hoc ergo propter hoc (“after the thing, therefore because of the thing”)’22 and exactly describes the uncontrolled before-and-after design.

The underlying logic of the uncontrolled before-and-after study makes it difficult to make reliable causal inferences about the impact of clinical information systems. Externally controlled before-and-after studies and the interrupted time series can both take into account known confounding factors but, like other observational designs, cannot make adjustments for unknown confounding; investigators therefore still need to consider the possibility of bias when interpreting results.23 In clinical medicine, the RCT is almost universally considered to be the gold standard for assessing cause and effect or therapeutic impact, if it is conducted and reported well and when it is practicable and warranted. It is the only design that effectively deals with the problem of unknown confounding and is therefore required for drug-licensing purposes.

Instead of rejecting RCTs, or alternatively of advocating their universal use, we argue there is a need for clarity about how to match study methods to evaluation questions (table 1). The method used should follow closely from the question. While an RCT is ideally suited to answer important questions about whether and by how much an information system improves clinical practice or patient outcomes, it is only necessary if the information system either costs a significant amount or may expose patients, professionals or health systems to added risk. For example, it would be inappropriate to use an RCT to evaluate the installation of an intranet for access to networked PCs, on the grounds of the low cost and low risk exposure. However, developing and delivering a regional decision support system can expose patients and professionals to the risk of worsened decisions.29 It might also expose the health system to increased resource use—for example, through increased test orders. Thus, an RCT to assess impact would be appropriate, especially if the question to be answered is: ‘Is the balance of benefit and harm due to this decision support system sufficient to mandate widespread use across a regional or national health system?’ By contrast, a before-and-after study would be adequate if stakeholders are interested in a descriptive answer to the question: ‘Is the new system installed in the local village family practice as fast as the old system in delivering test results?’

Table 1
Study designs to answer different questions about clinical information systems

Common concerns about the use of RCTs in medical informatics

What, then, are the concerns of the RCT skeptics? A selection of common concerns about the use of RCTs is listed in box 1 and discussed below. Many of these concerns are not unique to the medical informatics discipline and have been raised, for example, by evaluators of public health interventions and other healthcare interventions, or adapted from sources in medical informatics applications.3–8 13 14 30 31

Box 1

Common concerns about the use of randomized controlled trials (RCTs) in medical informatics

  1. Trials are unethical.
  2. Clinical information systems are too complex to be evaluated by RCTs.
  3. There is mixed evidence that RCTs can be successfully applied to medical informatics.
  4. Other study designs can also provide evidence that is just as reliable as RCTs.
  5. Trials are too expensive.
  6. Theory or case studies can reliably predict what works.
  7. Clinical information systems can do no harm.
  8. RCTs take too long, and technology in medical informatics moves too quickly.
  9. Trials answer questions that are not of interest to medical informatics.

Trials are unethical

The ethics of trials in medical informatics has been questioned.4 6 7 13 For example, in a 2009 evaluation of Scotland's National Telecare Development Programme, carried out by the York Health Economics Consortium, the RCT was rejected as an evaluation method for ethical reasons, ‘for example, deliberately withholding a service with known benefits.’13 The evidence is by no means unequivocal that telecare interventions are beneficial.32–34

The concern about trials being unethical is only true if the prior evidence that the health technology has more benefits than harms is overwhelming. For example, in 1941, the benefits of penicillin for very sick patients were so huge that there was no need for an RCT to be carried out.35 Unfortunately, the vast majority of therapies and drugs show only moderate or small effects, in which case a well-designed and conducted RCT is the only method that can provide unbiased estimates of their impacts.36 It should be noted that the use of treatments or clinical information systems that are not beneficial or even harmful is also unethical. Many have argued that, in the presence of uncertainty, it is unethical not to perform an RCT.30 37–41

Clinical information systems are too complex to be evaluated by RCTs

In the past, some public health researchers have asserted that the real world is too context-rich and chaotic for trials,30 a claim also made by many medical informatics researchers.3 4 6–8 12 The clinical world in which RCTs are routinely carried out is also messy and chaotic. Patients vary hugely in their severity of disease and concomitant diseases, their metabolism of and responsiveness to therapy and expression of symptoms. Actually, numerous complex interventions in many areas of healthcare have been successfully evaluated using the RCT. These include health-promotion programmes,42 behavioral interventions,43 educational visits,44 psychological debriefing,45 and other unusual non-drug interventions (table 2).

Table 2
Summary of 10 randomized controlled trials of non-drug technologies

Many of the social sciences (eg, educational research, psychology, and sociology) have a rich history of using RCTs.52 As shown in a classic 1980s RCT of a decision-support system for acute abdominal pain,53 a multiarm randomized trial can be used to quantify the contributions of different components of a complex intervention.14 15 54 Each arm of a multiarm trial refers to a group that is exposed to an intervention component or the control group. If the underlying components of a complex intervention are not known, a qualitative study such as an expert panel can sometimes be used to elicit the potentially important components.14 A multiarm RCT would then be needed to quantify the relative impacts of each of these components on patient outcomes. RCTs in medical informatics face the challenge of identifying valid, measurable, and quantifiable endpoints. The importance of measurement studies to design and validate such endpoints, in particular composite scores, is often overlooked, although methods to conduct such studies are well developed.14 55

RCTs can answer a subset of the many important questions that can be asked about a complex clinical information system, hence the need for a diversity of methods in evaluation.14

There is mixed evidence that RCTs can be successfully applied to medical informatics

Some informatics researchers express doubt as to whether RCTs can successfully assess the efficacy of clinical information systems. They have commented on (1) the low volume of experimental research in this field,56 (2) the poor quality of the experimental evidence to date,56 57 (3) where this evidence is available, the negligible overall impact on patient level outcomes,4 56 57 (4) the limited number of outcomes that a single RCT can assess,4 7 8 and (5) the tendency of experiments to underestimate benefits ‘because they define their outcomes too narrowly or take too short a time frame.’30

Several systematic reviews of RCTs of clinical information and computer-based decision support systems have been conducted.58–61 Balas et al reviewed 98 RCTs on the clinical value of computerized information services.59 Garg et al's review covered the impact of computer-based clinical-decision-support systems over the 1974–2004 period, from which 100 RCTs were identified.62 There has been a steady output of RCTs in the leading journals in recent years.63–65 Experimental evidence is not sparse but is unevenly distributed within the field of informatics. In a comprehensive search of more than 20 000 abstracts and titles in a systematic review of decision support for acute abdominal pain, only one RCT was found.21

Although only 13% of trials in Garg et al's systematic review demonstrated an improvement in patient outcomes, 64% of them showed an improvement in process-level outcomes (practitioner performance).62 The mean methodological quality score of identified trials, which showed an improvement over time, was a respectable 7 out of 10, although the range was wide (2 to 10). The variable quality of the RCTs in Garg et al's review is consistent with that of RCTs in the general clinical literature.66

Table 3 and figure 1 summarize the results of a MEDLINE search for RCTs in various areas of healthcare over 40 years.

Table 3
Percentage of evaluations that are randomized controlled trials in various health-related fields over time
Figure 1
Percentage of evaluations that are randomized controlled trials (RCTs) in various health-related fields over time.

The percentage of evaluations in traditional Chinese medicine that are RCTs increased from 0% in 1971–1975 to 62% in 2006–2010, with steep increases from the 1991–1995 quinquennial period onwards. There has been a move away from a theory-driven and mechanism-based approach to an evidence-based approach in evaluating interventions in traditional Chinese medicine,41 67 This may partly explain the huge relative increase in RCTs in this area. The percentages for dentistry and psychology also began to increase substantially in the 1991–1995 period, although the trends are not as dramatic as for traditional Chinese medicine. The trends for medical informatics are mixed, with steady increases from 1971–1975 to 1991–1995 and a fall thereafter. However, the absolute number of RCTs has increased in every quinquennial period since 1976–1980. The percentage of evaluations in informatics that are RCTs is the lowest among the four healthcare domains. There could be a number of reasons for this. For example, some questions on how and why informatics applications work can only be answered by conducting evaluations using other study designs, not the RCT. Another possible reason is that RCTs are not as widely accepted in medical informatics as in other fields of healthcare.

As mentioned earlier, some have expressed concerns that RCTs can only assess a limited number of outcomes, often those that can be measured most easily.4 7 8 14 Other designs can be used to complement RCTs to answer study questions that address a wide range of outcomes (see table 1).14

Some have also expressed concerns that RCTs tend to underestimate benefits, because of the narrow definition of their outcomes and their short-term time frame.30 There is clear empirical evidence that other study designs often overestimate benefits.40 68 If small benefits are useful, a systematic review of all past studies or a large primary study could have sufficient statistical power to detect these small effects. If the size of measured benefits is small, the RCT skeptic should consider the likely possibility that the trial is actually providing the correct answer.

Other study designs can also provide evidence that is just as reliable as RCTs

Although RCTs can be successfully applied in evaluating clinical information systems,62 Kaplan has rightly pointed out the need for ‘methodological pluralism.’5

There are many questions that can be asked about clinical information systems, some of which are listed in table 1, and indeed numerous methods to answer them. The most appropriate study design to answer many of these questions is not the RCT (table 1). We are however saying the RCT is the best design for obtaining unbiased estimates of the presence and size of small to moderate impacts that can be reliably attributed to the intervention and measured quantitatively. RCTs are sometimes too difficult or not feasible to carry out, so well-designed and well-conducted non-randomized studies are the only viable alternatives to quantitatively measure impact.14 Sections III and IV below provide a more detailed discussion of this topic.

Trials are too expensive

One reason RCTs in medical informatics are often considered infeasible is because they can be extremely expensive.4 7 Non-randomized studies can be costly as well—for example, a massive but inconclusive non-randomized study of over 15 000 patients who hoped to examine the benefits of using the Leeds AAP system, a decision-support system for acute abdominal pain.69 It is often assumed that RCTs are expensive, but the costs are partly determined by the study question, planning, and design. An RCT of educational visits in 25 obstetric units (n=4508 pregnant women) costed less than $60 000 to carry out.44 Costs of RCTs can be reduced by a number of methods, such as (1) narrowing the study focus to the most important questions, (2) canceling or postponing auxiliary studies, and (3) using routine data or employing more efficient methods to collect data (eg, via the internet). The RCT skeptic should also consider the ethical implications and costs to society and patients of conducting small, underpowered, and often inconclusive or misleading trials, and then approving technologies for routine use that are actually harmful, wasteful, and/or ineffective. In some instances, a meta-analysis of RCTs has been conducted to produce more precise estimates of effect, but this is not always possible.32 70–72 It is also true that some RCTs are unavoidably expensive, and investigators often face budgetary and logistical constraints. Less costly study designs should therefore be considered.

Theory or case studies can reliably predict what works

In recent years, theory-based or theory-generating approaches in evaluating clinical information systems have become more common.5 73–79 While these approaches are innovative and promising in explaining how, why, and in what context informatics applications work, it is too early to tell whether a better theoretical understanding will lead to improvements in system implementation, adoption, process-level outcomes, and patient outcomes.

Like other healthcare disciplines, there is a need in medical informatics to study underlying mechanisms and theories, characterize problems, and elicit underlying causal pathways, for which there are already many established methods.14 However, the primary concern to healthcare systems and policy makers (eg, National Institute for Health and Clinical Excellence: http://www.nice.org.uk/) is whether a treatment or health technology works. An understanding of the theoretical underpinnings of an informatics intervention does not entail it would work.

There are numerous examples from medicine where extensive laboratory research, apparently strengthened understanding of the disease and drug mechanisms, strongly suggested that a treatment should work and convinced skeptical ethics committees of the value of a clinical study, but failed to demonstrate a beneficial effect when put to the test of the RCT. Examples include:

  • the effects of corticosteroids on mortality of patients with head injuries80;
  • antivirals in the treatment of Bell's palsy patients81;
  • anticonvulsants in preventing deaths from pre-eclampsia82;
  • dietary supplementation with β carotene on prevention of cancer and heart disease.83

The arguments are the same for non-drug interventions (see table 2). Wynder provided many examples of discoveries that were shown to be of practical benefit long before the underlying mechanisms were known.31

Clinical information systems can do no harm

Karsh et al recently pointed out that ‘many [system] designers and policy makers believe that the risks of HIT [Health Information Technology] are minor and easily manageable,’ but in reality their belief is not warranted.84 The authors called this ‘the ‘Risk Free HIT’ fallacy.’ In the rush to roll out clinical information systems, risk assessments of the potential harms of technology to patients are often overlooked.9

Examples of medical informatics applications that pose potential or actual harm have been documented1 85 86 (also see http://iig.umit.at/efmi/badinformatics.htm). Legislation to regulate these applications have been proposed.87 In a study of errors in computerized physician order entry systems, a commonly used system was evaluated and unexpectedly found to increase the risk of 22 types of medication errors.1 In a case report on bar-coding, a diabetic hospital patient was wrongly given another patient's bar-coded identification wristband, with potentially fatal consequences.88 Routine data from electronic health record systems with embedded decision support can help detect harms to patients and decrease healthcare errors, but these systems can also introduce unintended risks. An important objective of clinical information systems is healthcare quality improvement, but these systems need to be made safer.89

RCTs take too long, and technology in medical informatics moves too quickly

RCTs can be time-consuming and labor-intensive.7 This could be an issue in evaluating rapidly developing IT applications. A recent study summarized some of the key changes in components of England's National Programme for IT over a short time frame.90

The duration of an RCT can sometimes be shortened, for example, by reducing the sample size needed through the employment of more sensitive measurements to assess outcome(s) if available, carrying out the trial in multiple centers, using crossover designs in which ‘measurements are made on the same participants with and without access to the information resource’14 when feasible, and including high-risk patients only in the trial, which can however reduce the generalizability of the trial results.14 26 54 However, these methods of shortening a trial can only be occasionally applied, because they are technically or logistically difficult to apply. If shortening the trial duration is not feasible, then investigators would need to decide on whether to use alternative study designs.

Non-randomized studies can also take a long time. For example, one of the authors of this paper (JLYL) was involved in a large-scale prospective cohort study of an intervention that took 5 years to complete.91 92 In fact, the evaluation was extended after 6 years, and new results only became available after 11 years.93 Even an uncontrolled before-and-after study can be time-consuming. One such study took several years to evaluate the impact of a clinical information system on patient care.94

Medical informatics is not the only field that moves quickly. Many areas of basic scientific research also develop rapidly with potentially important applications, but the translation of promising research into useful clinical tools is usually a long and arduous process, and much like medical informatics, the success rate is very low.95 96

Trials answer questions that are not of interest to medical informatics

RCTs cannot answer questions about how or why a clinical information system works, as critics correctly point out.3 4 7 13 Contrary to their claims, however, RCTs have sound logical foundations97 98 and can answer the following important questions:

  • Does doing A cause benefit X? How large is benefit X?
  • How often is harm Y caused by doing A?
  • Does doing A benefit group P more than group Q?
  • How much of the observed differences in X is caused by A? In other words, is A a marker for differences in X?
  • Does doing A cause the same benefit or harm as doing B? (relevant if A is cheaper, or more acceptable than B)

As in clinical medicine, many hypotheses of interest in medical informatics can be expressed as one of the above questions, and are thus amenable to a rigorous examination using the family of RCT methods.26 54

The RCT plays an important role in translating research into clinical practice. For the few promising clinical applications that survive to the evaluation stage, the RCT plays an integral part in deciding whether the final steps in the translation process are successful. Indeed, for health regulators such as the National Institute for Health and Clinical Excellence in England, such trials form an important component of the evidence base to support decisions about whether the entire National Health System will approve and purchase clinical information systems in the future.

Challenges to conducting RCTs in medical informatics

Although we have argued in favor of using RCTs to evaluate the impact of interventions in medical informatics, we also recognize the challenges of conducting randomized trials in this field. We explore a selection of these challenges below to illustrate some of the difficulties.

Feasibility of randomizing at the level of the individual patient

In informatics, when individual patients are randomly allocated to an intervention (eg, doctor's use of an information system) or a control (conventional care), doctors who used the system for patients in the experimental group might also have taken care of patients in the control group. The doctors might remember the output from the system and unintentionally use the information when caring for control group patients. This carryover effect can contaminate the control group.14 99–101 The estimated impact of the information system may be understated as a result.

Randomizing at the level of clusters

A trial design which avoids this problem randomizes at the level of ‘clusters,’ that is the healthcare provider, the department, or the hospital. Clusters are randomly allocated to the intervention or control group, not the individual patient. This type of RCT is called the cluster RCT.99–102 In informatics, cluster trials are often carried out successfully with useful results in diverse areas.64 103 104 However, randomizing at the patient level or the healthcare provider level can be achieved with only some interventions. In other instances, entire departments or hospitals may need to be randomized, requiring huge samples and considerable resources (in terms of cost, duration, logistics, and effort), which can make a cluster RCT infeasible.

Problem of equipoise

When strong evidence exists to suggest that a specific clinical information system is likely to be beneficial for patients, an RCT may not be feasible because the criterion of equipoise is not met. Equipoise refers to the principle that an eligible participant can be recruited to an RCT if there is genuine uncertainty in the informatics and clinical community on the net benefits of the new system for prospective participants.38 105 106

Roll-out and implementation of a clinical information system

When a clinical information system is being sequentially rolled out to individual patients or clusters, a conventional RCT often cannot be done because the equipoise principle is violated.107 108 To overcome this obstacle, a stepped wedge randomized design could be used, in which participants are randomly allocated to the intervention group (ie, sites or individuals that have received the intervention) or the control group (ie, sites or individuals that have not yet received the intervention), as the system is being rolled out sequentially.107 108

However, implementation of clinical information systems can be a highly complex process. Workflows might be more complicated than anticipated, and the intervention itself might change as the systems are being rolled out. This is the case with an ongoing prospective England-wide evaluation of implementation and adoption of electronic health records in secondary care.90 A stepped wedge randomized trial might not have been feasible in this instance.

Need for alternatives to RCTs

We have outlined above some of the challenges in designing and carrying out RCTs in medical informatics, which may partly explain the relatively slow adoption of RCTs in the field compared to others (table 3, figure 1). Large trials are sometimes infeasible, not the right study design to answer certain types of questions, or are often prohibitively expensive. Recruitment rates of prospective trial participants might be low because they might not be willing to accept randomized assignments. When RCTs are difficult or not feasible, there is a need to consider good-quality alternatives.

Some potential alternative study designs to RCTs

There are other study designs that might answer the question: ‘What would have happened if we had not introduced the information system?’ These designs include controlled before-and-after studies, interrupted time series and other observational designs.14 109 Time-series studies are often carried out when an appropriate control group cannot be identified and ‘attempt to detect whether an intervention has had an effect significantly greater than the underlying trend. Data are collected at multiple time points before and after the intervention.’109

A within-health professional before-and-after study (eg, Friedman et al29) evaluated the impact of extra information such as advice on simulated decisions on clinicians' diagnostic reasoning. Although well conducted, investigators and users of this controlled before-and-after evaluation need to consider the possibility of biases such as the ‘second look’ bias, that is, where else might health professionals have looked for information and how often would they actually use the test system in practice?110 111

An interrupted time-series study of clinical guidelines that are part of an electronic medical record suggested little impact on the process and cost of care but improvement in essential documentation.112 The authors pointed out that while their study was well implemented, their findings could potentially be biased from several sources, including temporal confounding.

Occasionally, a well-calibrated clinical prediction rule can be based on observational data when randomized trial data are not available, for example from the Duke cardiovascular registry, as long as it has been shown to perform well in the target group of participants.113 114

Conclusion

RCTs are neither a perfect nor universal evaluation method, but form an integral part of a medical informatics evaluator's toolkit, hence the need for such studies. RCTs are not being advocated as the only method for evaluation, but there is a need for various approaches in which different approaches and study designs are used to answer different questions and to address different aspects of a given system. The RCT is an important and powerful method, and an underused one in medical informatics.21 Wyatt and Friedman's text on evaluation methods in medical informatics covers a wide range of methods including both quantitative and qualitative methods, precisely because no single study design alone can be used to answer all the possible questions about a clinical information system.14

In conclusion, we addressed a selection of common concerns about RCTs and examined the extent to which they are fallacious. We believe there is a need for guidelines concerning the conduct of impact studies in clinical information systems and are working to draft these in collaboration with the CONSORT group.21 28

Acknowledgments

We would like to thank DG Altman for comments on earlier versions of this manuscript.

Footnotes

Funding: JLYL was supported by The Scottish Funding Council for this work.

Competing interests: None.

Contributed by

Contributor: JLYL had the idea for this paper and wrote this manuscript with contributions from JCW.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203 [PubMed]
2. Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics. J Am Med Inform Assoc 1994;1:353–5 [PMC free article] [PubMed]
3. Heathfield HA, Buchan IB. Current evaluations of information technology in health care are often inadequate. BMJ 1996;313:1008. [PMC free article] [PubMed]
4. Heathfield HA, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998;316:1959–61 [PMC free article] [PubMed]
5. Kaplan B. Evaluating informatics applications—some alternative approaches, and call for methodological pluralism. Int J Med Inform 2001;64:39–56 [PubMed]
6. UK Institute of Health Informatics Evaluation of Electronic Patient and Health Record Projects. Winchester: ERDIP Programme, NHS Information Authority, 2001
7. Stoop A, Heathfield H, de Mul M, et al. Evaluation of patient care information systems: theory and practice. In: Berg M, editor. , ed. Health Information Management: Integrating Information Technology in Health Care Work. London: Routledge, 2004:206–29
8. Brender J. Handbook of Evaluation Methods for Health Informatics. London: Elsevier Academic Press, 2006
9. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systematic evaluation. PLoS Med 2009;6:e10000126 [PMC free article] [PubMed]
10. Rigby M. Health informatics as a tool to improve quality in non-acute care—new opportunities and a matching need for a new evaluation paradigm. Int J Med Inform 1999;56:141–50 [PubMed]
11. Patel VL, Kaufmann DR, Arocha JF. Emerging paradigms of cognition in medical decision making. J Biomed Inform 2002;35:52–75 [PubMed]
12. Kaplan B, Shaw NT. People, Organizational, and Social Issues. IMIA Yearbook of Medical Informatics. Heidelberg: Schattauer, 2002:91–102
13. Beale S, Sanderson D, Kruger J, for the York Health Economics Consortium Evaluation of the Telecare Development Programme: Final Report. Edinburgh: Scottish Government, 2009
14. Friedman CP, Wyatt JC. Evaluation Methods in Bio-Medical Informatics. 2nd edn New York: Springer-Verlag, 2006
15. Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall, 1991
16. Pope C, Mays N. Qualitative Research in Health Care. 2nd edn London: BMJ Publishing, 2000
17. Rothman KJ, Greenland S. Modern Epidemiology. 2nd edn Philadelphia: Lippincott-Raven, 1998
18. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13:16–23 [PMC free article] [PubMed]
19. de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 2008;77:41–9 [PubMed]
20. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 2009;16:531–8 [PMC free article] [PubMed]
21. Liu JLY, Wyatt JC, Deeks JJ, et al. Systematic reviews of clinical decision tools for acute abdominal pain. Health Technol Assess 2006;10:1–186, iii–iv. [PubMed]
22. Copi IM, Cohen C. Introduction to Logic. 9th edn Englewood Cliffs, NJ: Prentice-Hall, 1994
23. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251–9 [PubMed]
24. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services. BMJ 1995;311:42–5 [PMC free article] [PubMed]
25. Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the Economic Evaluation of Health Care Programmes. 3rd edn New York: Oxford University Press, 2005
26. Pocock SJ. Clinical Trials: A Practical Approach. Chichester: John Wiley and Sons, 1983
27. Schulz KF, Altman DG, Moher D, the CONSORT Group CONSORT 2010 Statement: updated guidelines for reporting parallel-group randomized trials. BMJ 2010;340:c332. [PMC free article] [PubMed]
28. Moher D, Hopewell S, Schulz KF, et al. The CONSORT Statement 2010 updated guidelines for reporting randomized trials: explanation and elaboration. BMJ 2010;340:c869. [PMC free article] [PubMed]
29. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians' diagnostic reasoning by computer-based consultations: a multisite study of 2 systems. JAMA 1999;282:1851–6 [PubMed]
30. Macintyre S, Petticrew M. Good intentions and received wisdom are not enough. J Epidemiol Community Health 2000;54:802–3 [PMC free article] [PubMed]
31. Wynder EL. Studies in mechanism and prevention: striking a proper balance. Am J Epidemiol 1994;139:547–9 [PubMed]
32. McLean S, Chandler D, Nurmatov U, et al. Telehealthcare for asthma. Cochrane Database Syst Rev 2010;(10):CD007717. [PubMed]
33. Liu JLY, Wyatt JC. The Impact of Telepsychiatry, Teledermatology, Teleent and Teleradiology on the Health Services: A Rapid Review. Portland, OR: Medicaid Evidence-Based Decisions Program, 2008
34. Clark JS, Mair FS, O'Donnell C, et al. E-health: implementation and evaluation research in Scotland—a scoping exercise. J Telemed Telecare 2008;14:119–21 [PubMed]
35. Abraham EP, Chain E, Fletcher CM. Further observations on penicillin. Lancet 1941;2:177–88
36. Peto R, Pike MC, Armitage P, et al. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design. Br J Cancer 1976;34:585–612 [PMC free article] [PubMed]
37. Collins R, Peto R, Gray R, et al. Large-Scale Randomized Evidence: Trials and Overviews. Oxford Textbook of Medicine. Oxford: Oxford University Press, 1998:21–32
38. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317:1170–1 [PMC free article] [PubMed]
39. Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet 2001;357:373–80 [PubMed]
40. Sacks H, Chalmers TC, Smith H., Jr Randomized versus historical controls for clinical trials. Am J Med 1982;72:233–40 [PubMed]
41. Tang JL, Leung PC. An efficacy-driven approach to the research and development of traditional Chinese medicine. Hong Kong Med J 2001;7:375–80 [PubMed]
42. Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study. Imperial Cancer Research Fund OXCHECK Study Group. BMJ 1995;310:1099–104 [PMC free article] [PubMed]
43. Proudfoot J, Guest D, Carson J, et al. Effect of cognitive-behavioural training on job-finding among long-term unemployed people. Lancet 1997;350:96–100 [PubMed]
44. Wyatt JC, Paterson-Brown S, Johanson R, et al. Trial of outreach visits to enhance the use of systematic reviews in 25 obstetric units. BMJ 1998;317:1041–6 [PMC free article] [PubMed]
45. Hobbs M, Mayou R, Harrison B, et al. A randomised controlled trial of psychological debriefing for victims of road traffic accidents. BMJ 1996;313:1438–9 [PMC free article] [PubMed]
46. Allen K, Blascovich J. The value of service dogs for people with severe ambulatory disabilities. A randomised controlled trial. JAMA 1996;275:1001–6 [PubMed]
47. Byrd RC. Positive therapeutic effects of intercessory prayer in a coronary care unit population. South Med J 1988;81:826–9 [PubMed]
48. Moses S, Manji F, Bradley JE, et al. Impact of user fees on attendance at a referral centre for sexually transmitted diseases in Kenya. Lancet 1992;340:463–6 [PubMed]
49. Kelly JA, Murphy DA, Sikkema KJ, et al. Randomised, controlled, community-level HIV-prevention intervention for sexual-risk behaviour among homosexual men in US cities. Community HIV Prevention Research Collaborative. Lancet 1997;350:1500–5 [PubMed]
50. Close J, Ellis M, Hooper R, et al. Prevention of falls in the elderly trial (PROFET): a randomised controlled trial. Lancet 1999;353:93–7 [PubMed]
51. Frasure-Smith N, Lesperance F, Prince RH, et al. Randomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction. Lancet 1997;350:473–9 [PubMed]
52. Oakley A. Experimentation and social interventions: a forgotten but important history. BMJ 1998;317:1239–42 [PMC free article] [PubMed]
53. Wellwood J, Johannesen S, Spiegelhalter DJ. How does computer aided diagnosis improve the managment of acute abdominal pain? Ann R Coll Surg Engl 1992;74:40–6 [PMC free article] [PubMed]
54. Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. New York: Springer Verlag, 1998
55. Friedman CP, Abbas UL. Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems. Int J Med Inform 2003;69:261–72 [PubMed]
56. Black A, Car J, Majeed A, et al. Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research. Inform Prim Care 2008;16:175–7 [PubMed]
57. Car J, Black A, Anandan C, et al. The Impact of Ehealth on the Quality and Safety of Healthcare: A Systematic Overview and Synthesis of the Literature. Birmingham: NHS Connecting for Health Evaluation Programme, 2008 [PMC free article] [PubMed]
58. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician perfomance and patient outcomes: a systematic review. JAMA 1998;280:1339–46 [PubMed]
59. Balas EA, Austin SM, Mitchel JA, et al. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996;5:271–8 [PubMed]
60. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med 1994;120:135–42 [PubMed]
61. Haynes RB, Walker CJ. Computer-aided quality assurance: a critical appraisal of research. Arch Intern Med 1987;147:1297–301 [PubMed]
62. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223–38 [PubMed]
63. Leung GM, Johnston JM, Tin KYT, et al. Randomised controlled trial of clinical decision tools to improve learning of evidence based medicine in medical students. BMJ 2003;327:1090. [PMC free article] [PubMed]
64. Goud R, de Keizer NF, ter Riet G, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabiltation. BMJ 2009;338:b1440. [PMC free article] [PubMed]
65. Swartz LHG, Noell JW, Schroeder SW, et al. A randomised control study of a fully automated internet based smoking cessation programme. Tob Control 2006;15:7–12 [PMC free article] [PubMed]
66. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365:1159–62 [PubMed]
67. Tang JL. Research priorities in traditional Chinese medicine. BMJ 2006;333:391–4 [PMC free article] [PubMed]
68. Deeks JJ, Dinnes J, D'Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:iii–x, 1–173. [PubMed]
69. Adams ID, Chan M, Clifford PC, et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ (Clin Res Ed) 1986;293:800–4 [PMC free article] [PubMed]
70. Murray E, Burns J, See ST, et al. Interactive health communication applications for people with chronic disease. Cochrane Database Syst Rev 2005;(4):CD004274. [PubMed]
71. Clark RA, Inglis SC, McAlister FA, et al. Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis. BMJ 2007;334:942–5 [PMC free article] [PubMed]
72. Guyatt G, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of an individual trial still matter? PLoS Med 2008;5:e4. [PMC free article] [PubMed]
73. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010;10:12. [PMC free article] [PubMed]
74. Greenhalgh T, Potts HW, Wong G, et al. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q 2009;87:729–88 [PMC free article] [PubMed]
75. Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak 2010;10:67. [PMC free article] [PubMed]
76. Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med 2009;6:e10000086 [PMC free article] [PubMed]
77. Pawson R. Evidence-based policy: the promise of ‘realist synthesis.’ Evaluation 2002;8:340–58
78. Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005;(10 Suppl 1):21–34 [PubMed]
79. Greenhalgh T, Stones R. Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory. Soc Sci Med 2010;70:1285–94 [PubMed]
80. Edwards P, Arango M, Balica L, et al. CRASH trial collaborators Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroids in adults with head injury–outcomes at 6 months. Lancet 2005;365:1957–9 [PubMed]
81. Sullivan FM, Swan IRC, Donnan PT, et al. Early treatment with prednisolone or acyclovir in Bell's Palsy. N Engl J Med 2007;357:1598–607 [PubMed]
82. Altman D, Carroli G, Duley L, et al. The Magpie Trial Collaborative Group Do women with pre-eclampsia, and theirr babies, benefit from magnesium sulphate? The Magpie Trial: a randomised placebo-controlled trial. Lancet 2002;359:1877–90 [PubMed]
83. Hennekens CH, Buring JE, Manson JE, et al. Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease. N Engl J Med 1996;334:1145–9 [PubMed]
84. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010;17:617–23 [PMC free article] [PubMed]
85. Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health informatics services. BMJ 2001;323:552–6 [PMC free article] [PubMed]
86. Roberts JM, Copeland KL. Clinical websites are currently dangerous to health. Int J Med Inform 2001;62:181–7 [PubMed]
87. Donawa M. FDA final guidance on software validation. Med Device Technol 2002;13:20–4 [PubMed]
88. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss. Ann Intern Med 2006;144:510–16 [PubMed]
89. Newton RC, Mytton OT, Aggarwal R, et al. Making existing technology safer in healthcare. Qual Saf Health Care 2010;19(Suppl 2):i15–24 [PubMed]
90. Robertson A, Cresswell K, Takian A, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ 2010;341:c4564. [PMC free article] [PubMed]
91. Ong SG, Liu J, Wong CM, et al. Studies on the respiratory health of primary school children in urban communities of Hong Kong. Sci Total Environ 1991;106:121–35 [PubMed]
92. Peters J, Hedley AJ, Wong CM, et al. Effects of an ambient air pollution intervention and environmental tobacco smoke on children's respiratory health in Hong Kong. Int J Epidemiol 1996;25:821–8 [PubMed]
93. Hedley AJ, Wong CM, Thach TQ, et al. Cardiorespiratory and all-cause mortality after restriction on sulphur content of fuel in Hong Kong: an intervention study. Lancet 2002;361:787–8 [PubMed]
94. Ho LM, McGhee SM, Hedley AJ, et al. The application of a computerized problem-oriented medical record system and its impact on patient care. Int J Med Inform 1999;55:47–59 [PubMed]
95. Contopoulos-Ioannidis DG, Ntzani EE, Ioannidis JP. Translation of highly promising basic science research into clinical applications. Am J Med 2003;114:477–84 [PubMed]
96. Crowley WF., Jr Translation of basic research into useful treatments: how often does it occur? Am J Med 2003;114:503–5 [PubMed]
97. Mill JS. System of Logic, Ratiocinative and Inductive. 5th edn London: Parker, Son and Bowin, 1862
98. Copi IM, Cohen C. Introduction to Logic. 13th edn Englewood Cliffs, NJ: Prentice-Hall, 2008
99. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ 2001;322:355–7 [PMC free article] [PubMed]
100. Puffer S, Torgerson DJ, Watson J. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ 2003;327:785–9 [PMC free article] [PubMed]
101. Gilbody S, Bower P, Torgerson D, et al. Cluster randomized trials produced similar results to individually randomized trials in a meta-analysis of enhanced care for depression. J Clin Epidemiol 2008;61:160–8 [PubMed]
102. Bennett S, Braunholtz D, Chinn S, et al. Cluster Randomised Trials: Methodological and Ethical Considerations. MRC Clinical Trials Series. London: Medical Research Council, 2002
103. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941–4 [PMC free article] [PubMed]
104. Eminovic N, de Keizer NF, Wyatt JC, et al. Teledermatologic consultation and reduction in referrals to dermatologists: a cluster randomized controlled trial. Arch Dermatol 2009;145:558–64 [PubMed]
105. Friedman B. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141–5 [PubMed]
106. Fries JF, Krishnan E. Equipoise, design bias and randomized controlled trials: the elusive ethics of new drug development. Arthritis Res Ther 2004;6:R250–5 [PMC free article] [PubMed]
107. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006;6:54. [PMC free article] [PubMed]
108. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007;28:182–91 [PubMed]
109. Grimshaw J, Campbell M, Eccles M, et al. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract 2000;(17 Suppl 1):S11–18 [PubMed]
110. Carey TS, Garrett J. Patterns of ordering diagnostic tests for patients with acute low back pain. The North Carolina Back Pain Project. Ann Intern Med 1996;125:807–14 [PubMed]
111. Jones TV, Gerrity MS, Earp J. Written case simulations: do they predict physicians' behaviour? J Clin Epidemiol 1990;43:805–15 [PubMed]
112. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effects on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7:186–95 [PMC free article] [PubMed]
113. Lee KL, Pryor DB, Harrell FE, Jr, et al. Predicting outcome in coronary disease: statistical models versus expert clinicians. Am J Med 1986;80:553–60 [PubMed]
114. Hlatky MA, Califf RM, Harrell FE, Jr, et al. Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery. J Am Coll Cardiol 1988;11:237–45 [PubMed]
115. Vall Mayans M, Hall AJ, Inskip HM, et al. Do bedbugs transmit hepatitis B? Lancet 1994;343:761–3 [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...