|
|
Health Serv Res. 2005 April; 40(2): 477–498. doi: 10.1111/j.1475-6773.2005.00368.x. | PMCID: PMC1361152 |
Copyright © 2005 Health Research and Education Trust. All rights reserved Can Computer-Generated Evidence-Based Care Suggestions Enhance Evidence-Based Management of Asthma and Chronic Obstructive Pulmonary Disease? A Randomized, Controlled Trial William M Tierney, J Marc Overhage, Michael D Murray, Lisa E Harris, Xiao-Hua Zhou, George J Eckert, Faye E Smith, Nancy Nienaber, Clement J McDonald, and Fredric D Wolinsky Objective Translation of evidence-based guidelines into clinical practice has been inconsistent. We performed a randomized, controlled trial of guideline-based care suggestions delivered to physicians when writing orders on computer workstations. Study Setting Inner-city academic general internal medicine practice. Study Design Randomized, controlled trial of 246 physicians (25 percent faculty general internists, 75 percent internal medicine residents) and 20 outpatient pharmacists. We enrolled 706 of their primary care patients with asthma or chronic obstructive pulmonary disease. Care suggestions concerning drugs and monitoring were delivered to a random half of the physicians and pharmacists when writing orders or filling prescriptions using computer workstations. A 2 × 2 factorial randomization of practice sessions and pharmacists resulted in four groups of patients: physician intervention, pharmacist intervention, both interventions, and controls. Data Extraction/Collection Methods Adherence to the guidelines and clinical activity was assessed using patients' electronic medical records. Health-related quality of life, medication adherence, and satisfaction with care were assessed using telephone questionnaires. Principal Findings During their year in the study, patients made an average of five scheduled primary care visits. There were no differences between groups in adherence to the care suggestions, generic or condition-specific quality of life, satisfaction with physicians or pharmacists, medication compliance, emergency department visits, or hospitalizations. Physicians receiving the intervention had significantly higher total health care costs. Physician attitudes toward guidelines were mixed. Conclusions Care suggestions shown to physicians and pharmacists on computer workstations had no effect on the delivery or outcomes of care for patients with reactive airways disease. Keywords: medical decision making, guidelines, quality improvement The Institute of Medicine has also stated that electronic medical record systems are “an essential technology for health care” ( Institute of Medicine, Committee on Improving the Medical Record 1991) that could improve medical practice ( Johnston et al. 1994; Tierney 2001). However, clinical information systems are expensive ( Dambro, Weiss, and McClure 1988), potentially intrusive ( Krall and Sittig 2001), and have not always improved care ( Johnston et al. 1994). We have previously shown that computer-based interventions can increase preventive care ( McDonald et al. 1984; Tierney, Hui, and McDonald 1986; McDonald et al. 1999; Tierney 2001) and reduce costs ( Tierney, Miller, and McDonald 1990; Tierney et al. 1993). We have had less success affecting chronic management of renal disease ( Harris et al. 1998) or heart disease ( Tierney et al. 2003; Subramanian et al. 2004). We assessed whether guideline-based care suggestions delivered via physicians' and pharmacists' computer workstations could improve the outpatient management and outcomes among patients with asthma or COPD. Setting and Subjects This study was approved by the University Institutional Review Board (study #9211-26) and took place in Indiana University Medical Group-Primary Care (IUMG-PC), an inner-city primary care practice-based research network ( Tierney et al. 1991). This study utilized four hospital-based academic practices where 25 faculty general internists and more than 100 internal medicine residents cared for approximately 13,000 patients during 50,000 annual visits. These practices have separate nursing and clerical staff but share paper and electronic medical records ( McDonald et al. 1999). At the beginning of each academic year, new physicians were randomly assigned to practices of departing physicians ( Tierney et al. 1991). Physicians were assigned to half-day sessions in a single practice: faculty attended 2–5 half-day sessions per week while residents attended one half-day session per week. Prior studies have shown no systematic differences in practice patterns or clinical outcomes between the four practices ( McDonald et al. 1984; Tierney, Miller, and McDonald 1990). Each physician cared for an assigned panel of patients. Faculty and residents practiced side by side. Residents briefly presented each patient to a faculty physician yet made all diagnostic and therapeutic decisions for their patients. This study also included outpatient pharmacists. We separately randomized the 11 full-time and 9 part-time pharmacists to intervention or control status. Patients were eligible if they were at least 18 years old, had either previously visited the study practices in the past year, and had either (1) the diagnosis of asthma or COPD recorded during any inpatient, emergency, or outpatient visit; (2) emphysema recorded as a reading on any prior chest radiograph or CT scan; or (3) two or more prescriptions for inhaled β-agonists, corticosteroids, ipratropium, or cromolyn, or oral β-agonists or theophylline. Prior studies ( Murray et al. 1995) and repeated internal audits have shown that 95 percent of patients visiting these study practices receive all of their medications from the hospital's outpatient pharmacy. Study Design and Randomization This study used a 2 × 2 factorial design to study the relative and additive effects of separate physician and pharmacist interventions. We randomized all half-day sessions to intervention or control status via a coin flip. Because faculty physicians practiced in more than one half-day session per week, this procedure resulted in a few physicians having sessions with both intervention and control sessions. A biostatistician blinded to their names switched sessions so that the number of intervention and control sessions per practice were equal and all physicians practiced only in sessions with the same study status. To minimize contamination, all physicians in each half-day session shared the same study status. Patients shared their assigned primary care physicians' study status. As patients were enrolled, they were randomly assigned by computer to receive all of their outpatient drugs from intervention or control pharmacists. Randomizing both practice sessions and patients resulted in four groups of patients: physician intervention only, pharmacist intervention only, both pharmacist and physician interventions, and no intervention (controls). Evidence-Based Care Suggestions The care suggestions were reviewed by a panel of local general internists, pulmonary specialists, and pharmacists. Each care suggestion displayed to the physician or pharmacist explained the rule with a link to the annotated guideline with references. Panel members could independently accept each rule as is, with minor modifications, or major modifications, or reject it. Rules with major modifications or rejected were adjudicated during an expert panel meeting. The final rules were programmed into physicians' computer workstations that have been in continuous use in this practice since 1984 ( McDonald et al. 1999). Care suggestions focused on: (1) performing pulmonary function tests, (2) giving influenza and pneumococcal vaccinations, (3) prescribing inhaled steroid preparations in patients with frequent symptoms of dyspnea, (4) prescribing inhaled anticholinergic agents in patients with COPD, (5) escalating doses of inhaled β-adrenergic agonists for all patients with persistent symptoms, (6) prescribing theophylline for patients with COPD and continued symptoms despite aggressive use of inhaled anticholinergic agents, β-agonists, and steroids, and (7) encouraging smoking cessation. (Leucotriene inhibitors did not become widely available until after this study ended.) The goal of this study was to assess the effects of patient-specific care suggestions, not their effects as an educational tool. Therefore, we presented the asthma and COPD guidelines to Medicine and Pharmacy Grand Rounds. We also met with all physicians and pharmacists and gave each a printed summary of the final asthma and COPD management guidelines that were the standard of care in this practice. The printout also listed the Consensus Panel members and summarized the process for developing the rules. Physician Intervention Protocol When patients presented for usual care, the physician received the patient's paper chart along with a computer-generated paper encounter form that included a list of active medications. Any study care suggestions for which a patient was eligible that day were printed at the bottom of the medication list. After the patient visit, the physician wrote orders on a workstation. No hand-written orders were allowed. Because patients' electronic medical records lacked symptom data and emergency and inpatient visits to outside hospitals, both intervention and control physicians were required to enter the severity of the subject's dyspnea (none, mild, moderate, severe) and number of exacerbations in the previous month. We required no further actions from control physicians, and no care suggestions were given although care suggestions were generated for all patients. Care suggestions were only presented to intervention physicians, both on their paper medication lists and on their computer workstations. The workstation displayed “suggested orders” on the main workstation screen (see online-only appendix at http://www.blackwellpublishing.com/products/journals/suppmat/HESR/HESR00369/HESR00369sm.htm). The physician was required to view all care suggestions (see online-only appendix at http://www.blackwellpublishing.com/products/journals/suppmat/HESR/HESR00369/HESR00369sm.htm), which contained three separate windows. The window at the bottom middle of the screen contained the actual suggested orders. The upper left window listed possible actions for each order (“order” or “omit”). The upper center window contained a short note explaining each care suggestion. Physicians could display the guideline and supporting references by hitting the “help” key. Patients could also present without appointments. If the patient's assigned physician was present, that physician usually cared for the patient and the intervention protocol was followed. In all other cases, the care suggestions were only displayed if both the patient and physician were assigned to intervention status. Throughout the study, all physicians received other messages, reminders, and suggested orders, mostly drug interaction warnings and preventive care suggestions not related to this study's target conditions ( McDonald et al. 1984). Pharmacist Intervention Protocol All prescriptions were sent electronically to the outpatient pharmacy computer system that printed labels for pill bottles. For all enrolled subjects, any study care suggestions were sent electronically to the outpatient pharmacy's Pharmacist Intervention Recording System ( Overhage and Lukes 1999) in which outpatient pharmacists recorded all prescription problems and interactions with patients. For patients randomized to the pharmacist intervention, instead of printing the pill bottle labels the system printed a message indicating that there were care suggestions that should only be viewed by an intervention pharmacist who could respond to them, if desired, and then fill the prescriptions. These suggestions were identical to those presented to the primary care physician. Care suggestions were only displayed when both the pharmacist and the patient were in the pharmacist intervention group. Notably, patients were eligible for the pharmacist intervention whenever they presented to the outpatient pharmacy, not just after primary care visits and not just with prescriptions for asthma or COPD. When viewing care suggestions, the pharmacist had three options: do nothing, discuss them with the patient, or discuss them with the physician. However, during the first 2–3 months of this study, physician discussions rarely occurred. Pharmacists found paging or calling physicians difficult and time consuming. We therefore added an electronic mail option that allowed intervention pharmacists to send care suggestion to the patient's physician, adding text as desired. The physician received a notice of the electronic mail message after logging onto any workstation but was not required to read or respond to it. Subject Enrollment and Data Collection We used data from patients' electronic medical records ( McDonald et al. 1999) to generate weekly lists of eligible patients with primary care appointments. A research assistant approached potential subjects in the practice's waiting room, escorted the patient to a private area, explained the study, and invited the patient to participate. Agreeable patients signed an informed consent statement and were scheduled for 45-minute telephone interviews to collect demographic information, income, educational level, and smoking history. At enrollment and during close-out 12 months later, we also administered the SF-36, a measure of health-related quality of life ( McHorney and Ware 1995). To assess disease-specific quality of life, we administered he McMaster Chronic Respiratory Disease Questionnaire (CRQ) ( Guyatt et al. 1987) for patients with COPD or the McMaster Asthma Quality-of-Life Questionnaire (AQLQ) ( Juniper et al. 1993). We also administered the American Board of Internal Medicine's patient satisfaction questionnaire ( Webster 1988), a measure of physicians' communication abilities that had been previously validated in this practice ( Dexter et al. 1996). We assessed pharmacy satisfaction with a locally developed questionnaire ( Weinberger et al. 2001). Medication adherence was assessed using surveys developed by Inui, Carter, and Pecoraro (1981) and Morisky, Green, and Levine (1986). We also assessed medication compliance with the medication possession ratio ( Steiner et al. 1988), which uses refill data from patients' pharmacy records and correlates well with other adherence measures ( Choo et al. 1999). A research assistant telephoned the patient at the beginning of his or her twelfth study month to administer the close-out interview. If no telephone interview could be held during the twelfth month, a research assistant attempted to interview the patient during visits made during the thirteenth and fourteenth months after enrollment. We extracted information from patients' electronic medical records, including evidence of compliance with the study care suggestions, hospitalizations, emergency department visits, and direct health care charges. To assess the degree to which physicians' attitudes toward guidelines may have affected their response to the intervention, we also administered a questionnaire to study physicians (after their Grand Rounds on guidelines but before the study started) that included a questionnaire created by Tunis et al. (1994). This questionnaire was originally developed with data from 1,513 generalist and specialist internists. It assessed personal attitudes toward guidelines; no correlations were made with practice behavior. Statistical Analyses The unit of analysis for all outcomes was the patient. Because we intervened on the physician and pharmacist, differences between study groups were assessed using random effects generalized linear models (for continuous variables) and generalized estimating equations (for categorical variables) to account for correlations of outcomes among patients treated by specific physicians. We did not adjust for the intervention pharmacist because there was no one-to-one assignment of pharmacists to patients. The primary outcome was adherence to the guideline-based care suggestions. If a patient were eligible for same care suggestion on multiple visits, it was considered as one care suggestion. Adherence occurred when the indicated action was taken any time after it was first suggested and before patient close-out. We used analysis of variance to compare adherence to the care suggestions between the four study groups and Student's t-tests to assess the separate effects of the physician and pharmacist interventions. We used Poisson's regression with an overdispersion parameter to compare the numbers of outpatient and emergency visits and hospitalizations between study groups. After creating overall and subscale scores for the SF-36, AQLQ, CRQ, and the physician and pharmacist satisfaction surveys, we used analysis of covariance to compare these scores between study groups, including baseline scores and indicators for ceiling and floor effects as covariates. We analyzed charges and medication possession ratios using Wald-type tests for log-normal data containing zeroes ( Zhou and Tu 1999). We used logistic regression to compare results of the Inui measure of medication adherence and analysis of covariance for the Morisky instrument, including baseline scores and indicators for ceiling and floor effects as covariates. Because our main focus was to improve patient-centered outcomes, data from a prior study ( Dexter et al. 1996) indicated that we needed 500 patients for 80 percent power to detect one unit change in the standard error of measurement for each subscale score of the SF-36 ( Wyrwich, Tierney, and Wolinsky 1999). Anticipating 25 percent attrition of subjects during the study, we sought to enroll 700 patients. Between January 1, 1994 and May 1, 1996, 1,730 potentially eligible subjects kept scheduled primary care visits (). Of these, 1,181 (63 percent) were contacted and agreed to be interviewed, 865 (73 percent) of whom agreed to participate and signed informed consent statements, and 706 (82 percent) completed baseline interviews, were enrolled, and were randomized. Of the 274 physicians practicing in the study practices during this 3-year study, 133 practiced in half-day sessions randomized to intervention status and 141 practiced in control sessions. A quarter of physicians were faculty general internists, the rest internal medicine, or medicine–pediatrics residents. There were no significant demographic or clinical differences between the four patient groups ( Table 1). Two-thirds were women, 60 percent were caucasian, and two-thirds had COPD rather than asthma. The mean age was 51 years. Approximately two-thirds of each group completed a 12-month interview. More than 80 percent of patients in each study group had at least one scheduled visit to their primary care physician during their year in the study (mean of almost five visits). The mean number of study care suggestions per patient was just under three. Table 2 shows the number of subjects eligible for each care suggestion and the fraction ordered during the study. There were no differences between the four study groups in either adherence to the care suggestions, combined or individually. Post hoc power analyses showed that it would take 145 subjects per group to have 80 percent power to detect a 50 percent increase in adherence to the care suggestions (from 32 percent to 48 percent, a 16 percent absolute increase) found in the control group. | Table 2Adherence to Treatment Guidelines |
There was also no consistent effect on quality of life ( Table 3). The only significant difference in any SF-36 subscale was a significantly improved Role Physical subscale in the group receiving both interventions. There were no differences in the CRQ subscales. For the AQLQ, patients with asthma receiving the pharmacist intervention had significantly improved scores in the emotion subscale. There were no intergroup differences in medication adherence or in patients' satisfaction with their physicians or pharmacists ( Table 3). Nor were there any differences in emergency department visits or hospitalizations for any cause or for reactive airways disease in particular ( Table 3). Patients in the group receiving only the physician intervention had significantly elevated total health care charges, possibly because of just a small number of extremely high-cost hospitalizations ( Table 3). Physicians' opinions of practice guidelines were mixed ( Table 4). They generally felt that guidelines were a good educational tool, a convenient source of information, and intended to improve the quality of care. Yet many physicians felt that guidelines represented oversimplified “cookbook” medicine, were often too rigid to apply to individual patients, hampered physician autonomy, and were used to decrease health care costs. | Table 4Physicians' Attitudes Toward Practice Guidelines |
Additional analyses were performed to ensure that the lack of significance was not because of the analytic methods chosen. Analyses of quality of life and satisfaction data were repeated after categorizing the changes from baseline into “better,”“no change,” and “worse” using the standard error of measurement ( Wyrwich, Tierney, and Wolinsky 1999). Direct health care charges, number of hospitalizations, and number of emergency department visits were also analyzed using Kruskal–Wallis nonparametric tests. These results of these analyses were similar to the primary analyses and are not presented. We had hoped to show that computer-generated care suggestions would enhance adherence to evidence-based guidelines and have salutatory effects on patient-centered and clinical outcomes. Unfortunately, none of this occurred, which surprised us, given our previous success with interventions to increase preventive care ( McDonald et al. 1984; Tierney, Hui, and McDonald 1986; McDonald et al. 1999) and reduce costs ( Tierney, Miller, and McDonald 1990; Tierney et al. 1993). Yet the workstations have been an effective medium for delivering interventions to affect physician ordering behavior ( Tierney et al. 1987; 1988; Tierney, Miller, and McDonald 1990). It is possible that providers will accept a computer's advice about preventive care and costs but may be less open to a computer's suggestions about managing chronic illnesses. A prior study of a similar computer-based intervention to improve physician adherence to local and national guidelines for heart disease was also ineffective ( Tierney et al. 2003). Another study used computer-generated reminders as part of a comprehensive management program for patients with chronic renal insufficiency. Despite mandatory consultation by nephrologists and a renal nurse, social worker, and dietician, there were no improvements in the processes or outcomes of care ( Harris et al. 1998). In fact, chronic disease management programs have had mixed effects in affecting the processes and outcomes of care ( Ferguson and Weinberger 1998). To date, the promise of computer-based decision support has not been realized. We have recently shown that including patient symptoms, their severity, and changes from the most recent primary care visit in patient-specific, computer-generated care suggestions had no effect on the processes or outcomes of care for veterans with heart failure ( Subramanian et al. 2004). Additional work is needed to improve the timing and content of such interventions. A recent review of computer-based decision-support systems found them to be effective in affecting preventive care but had less effect in chronic disease management and little evidence of improvement in patient outcomes ( Mitchell and Sulllivan 2001). The content, timing, and presentation of such decision-support tools need to be improved. Expertise in human factor engineering ( Norman 2002; Bates et al. 2003) may be particularly useful in designing interventions that physicians can and will use to improve the translation of evidence-based guidelines into everyday clinical practice. Although physicians and pharmacists had to view each suggestion, they could easily erase them from the screen by hitting the “escape” key. In a busy practice, the care suggestions could have been seen as a nuisance. Automatically deleting them may have become routine. We have strong evidence from two prior studies that this may have been the case. We found no effects of preventive care suggestions presented to residents on Wishard Hospital's inpatient medicine service ( Overhage, Tierney, and McDonald 1996). The same intervention was repeated on the same service several years later by Dexter et al. (2001), the only major difference being that the “escape” key was disabled. Each care suggestion had to be dealt with in some way, either by complying with it or explaining why it was inappropriate. The effects were dramatic: adherence increased by a factor of 45 over the rate found by Overhage. The key, then, is to find a balance between the hassle factor of an intervention that cannot be avoided and the patient benefits of adhering to evidence-based practice guidelines. It would seem to require clinician leadership to find the correct balance, or providers themselves could decide which care suggestions should require their input and response. The majority of patients in this study were middle-aged women with COPD, as defined using standard criteria. Yet the line between asthma and COPD among aging adults is blurred, and many patients with COPD have a reactive component to their lung disease that is amenable to pharmacotherapy. The strongest evidence for use of stepped care is among patients with asthma, which were in the minority in this study. This may have tempered providers' enthusiasm for following the care suggestions. Although pharmacists could not directly respond to care suggestions, they could have discussed them with the patients and/or physicians. In a work-sampling study performed during our intervention ( Murray et al. 1999), Murray et al. found that intervention pharmacists spent significantly more time than control pharmacists discussing information with patients, advising and informing them, and solving problems. Intervention pharmacists spent significantly less time checking and filling prescriptions and working alone, and they spent significantly more time than control pharmacists interacting with other pharmacy personnel, patients, and physicians and nurses. Hence, there were dramatic effects of the intervention on pharmacist–patient interactions that did not impact on patient care, outcomes, or satisfaction. Communication between physicians and pharmacists in this study was practically nonexistent. This is unfortunate, because as time for clinical encounters continues to shrink, these two groups of providers could have complementary roles in enhancing pharmacotherapy and monitoring for adverse outcomes ( Singhal, Raisch, and Gupchup 1999). Further work is needed to increase the collaboration of physicians and pharmacists and to define the role of computer-based interventions for improving the quality and outcomes of patient care. Although the Institute of Medicine believes electronic medical records are essential to improving the quality of care ( Institute of Medicine, Committee on Improving the Medical Record 1991; Institute of Medicine 2001), expensive and sometimes intrusive innovations need to be thoroughly tested in rigorous trials before broad implementation. Electronic medical record systems should be useful in enhancing evidence-based practice. However, in this study, providing patient-specific care suggestions for asthma and COPD management did not improve decision making or patient outcomes and cannot be recommended at this time. Additional research, especially with respect to chronic disease management, is required if computer information systems are to help bridge the quality chasm ( Institute of Medicine 2001). This work was supported by grant number HS07763 from the Agency for Healthcare Research and Quality. We thank the physicians, nurses, clerks, and administration of the IU Medical Group-Primary Care for their continued support of clinical research in general and this project in particular. - Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M, Middleton B. “Ten Commandments for Effective Decision Support: Making the Practice of Evidence-Based Medicine a Reality.” Journal of the American Medical Informatics Association. 2003;10(6):523–30. [PubMed]
- Cabana MD, Rand CS, Becher OJ, Rubin HR. “Reasons for Pediatrician Nonadherence to Asthma Guidelines.” Archives of Pediatric and Adolescent Medicine. 2001;155(9):1057–62.
- Canadian Thoracic Society Workshop Group. “Guidelines for the Assessment and Management of Chronic Obstructive Pulmonary Disease.” Canadian Medical Association Journal. 1992;147(4):420–8. [PubMed]
- Choo P, Rand C, Inui T, Lee ML, Cain E, Codiero-Breault M, Canning C, Platt R. “Validation of Patient Reports, Automated Pharmacy Records, and Pill Counts with Electronic Monitoring of Adherence to Antihypertensive Therapy.” Medical Care. 1999;37(9):846–57. [PubMed]
- Dambro MR, Weiss BD, McClure CL. “An Unsuccessful Experience with Computerized Medical Records in an Academic Medical Center.” Journal of Medical Education. 1988;63(8):617–23. [PubMed]
- Dexter PR, Perkins S, Overhage JM, Meharry K, Kohler RB, McDonald CJ. “A Computerized Reminder System to Increase the Use of Preventive Care for Hospitalized Patients.” New England Journal of Medicine. 2001;345(13):965–70. [PubMed]
- Dexter PR, Stump TE, Tierney WM, Wolinsky FD. “The Psychometric Properties of the SF-36 Health Survey among Older Adults in the Clinical Setting.” Journal of Clinical Gerophsychology. 1996;2(3):225–37.
- Ferguson JA, Weinberger M. “Case Management Programs in Primary Care.” Journal of General Internal Medicine. 1998;13(2):123–6. [PubMed]
- Guyatt GH, Berman LB, Townsend M, Pugsley SO, Chambers LW. “A Measure of Quality of Life for Clinical Trials in Chronic Lung Disease.” Thorax. 1987;42(10):773–8. [PubMed]
- Harris LE, Luft FC, Rudy DW, Kesterson JG, Tierney WM. “Effects of Multidisciplinary Case Management of Patients with Chronic Renal Insufficiency.” American Journal of Medicine. 1998;105(6):464–71. [PubMed]
- Institute of Medicine, Committee on Improving the Medical Record. The Computer-based Patient Record: An Essential Technology for Health Care. Washington, DC: National Academy Press; 1991.
- Institute of Medicine. Crossing the Quality Chasm: A New System for the 21st Century. Washington, DC: National Academy Press; 2001.
- Inui TS, Carter WB, Pecoraro RE. “Screening for Noncompliance among Patients with Hypertension: Is Self-Report the Best Available Measure?” Medical Care. 1981;19(10):1061–4. [PubMed]
- Johnston ME, Langton KB, Haynes RB, Mathieu A. “Effects of Computer-Based Clinical Decision Support Systems on Clinician Performance and Patient Outcomes.” Annals of Internal Medicine. 1994;120(2):135–42. [PubMed]
- Juniper E, Guyatt GH, Ferrie PJ, Griffith LE. “Measuring Quality of Life in Asthma.” American Review of Respiratory Diseases. 1993;147(4):832–8. [PubMed]
- Krall MA, Sittig DF. “Subjective Assessment of Usefulness and Appropriate Presentation Mode of Alerts and Reminders in the Outpatient Setting.” Proceedings of the American Medical Informatics Association Symposium. 2001:334–8.
- McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, McCabe GP. “Reminders to Physicians from an Introspective Computer Medical Record. A Two-Year Randomized Trial.” Annals of Internal Medicine. 1984;100(1):130–8. [PubMed]
- McDonald CJ, Overhage JM, Tierney WM, Dexter PR, Martin DK, Suico JG, Zafar A, Schadow G, Blevins L, Glazener T, Meeks-Johnson J, Lemmon L, Warvel J, Porterfield B, Warvel J, Cassidy P, Lindbergh D, Belsito A, Tucker M, Williams B, Wodniak C. “The Regenstrief Medical Record System: A Quarter Century Experience.” International Journal of Medical Informatics. 1999;54(3):225–53. [PubMed]
- McFadden ER, Gilbert IA. “Asthma.” New England Journal of Medicine. 1992;327(27):1928–37. [PubMed]
- McHorney CA, Ware JE., Jr “Construction and Validation of an Alternate form General Mental Health Scale for the Medical Outcomes Study Short-Form 36-Item Health Survey.” Medical Care. 1995;33(1):15–28. [PubMed]
- McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. “Promoting Cancer Prevention Activities by Primary Care Physicians: Results of a Randomized, Controlled Trial.” Journal of the American Medical Association. 1991;266(4):538–44. [PubMed]
- Mitchell E, Sulllivan F. “A Descriptive Feast but an Evaluative Famine: Systematic Review of Published Articles on Primary Care Computing during 1980–97.” British Medical Journal. 2001;322(7281):279–82. [PubMed]
- Morisky DE, Green LW, Levine DM. “Concurrent and Predictive Validity of a Self-Reported Measure of Medication Adherence.” Medical Care. 1986;24(1):67–74. [PubMed]
- Murray MD, Loos B, Wanzhu T, Eckert GJ, Zhou XH, Tierney WM. “Work Patterns of Ambulatory Care Pharmacists with Access to Electronic Guideline-Based Treatment Suggestions.” American Journal of Health-System Pharmacy. 1999;56(3):225–32. [PubMed]
- Murray MD, Rupp MT, Overhage JM, Ebbeler D, Main JW, Tierney WM. “Multidimensional Work Sampling in an Outpatient Pharmacy.” Pharmacy Practice Management Quarterly. 1995;15(3):44–56. [PubMed]
- National Asthma Education Program Expert Panel Report. Executive Summary: Guidelines for the Diagnosis and Management of Asthma. Washington, DC: 1994. U.S. Department of Health and Human Services (AHCPR Publication No. 94-3042A).
- Norman DA. The Design of Everyday Things. New York: Basic Books (Perseus); 2002.
- Overhage JM, Lukes A. “Practical, Reliable, Comprehensive Method for Characterizing Pharmacists' Clinical Activities.” American Journal of Health-System Pharmacy. 1999;56(23):2444–50. [PubMed]
- Overhage JM, Tierney WM, McDonald CJ. “Computer Reminders to Implement Preventive Care Guidelines for Hospitalized Patients.” Archives of Internal Medicine. 1996;156(14):1551–6. [PubMed]
- Shea S, DuMouchel W, Bahamonde L. “A Meta-Analysis of 16 Randomized Controlled Trials to Evaluate Computer-Based Clinical Reminder Systems for Preventive Care in the Ambulatory Setting.” Journal of the American Medical Informatics Association. 1996;3(6):399–409. [PubMed]
- Singhal PK, Raisch DW, Gupchup GV. “The Impact of Pharmaceutical Services in Community and Ambulatory Care Settings: Evidence and Recommendations for Future Research.” Annals of Pharmacotherapy. 1999;33(12):1336–55. [PubMed]
- Steiner JF, Koepsell TD, Fihn SD, Inui TS. “A General Method of Compliance Assessment Using Centralized Pharmacy Records: Description and Validation.” Medical Care. 1988;26(8):814–23. [PubMed]
- Subramanian U, Fihn SD, Weinberger M, Plue L, Smith FE, Udris E, McDonnel MB, Eckert GJ, Temkit M, Zhou XH, Chen L, Tierney WM. “A Controlled Trial of Including Symptom Data in Computer-Based Care Suggestions for Managing Chronic Heart Failure.” American Journal of Medicine. 2004;116(6):375–84. [PubMed]
- Tierney WM, Hui SL, McDonald CJ. “Delayed Feedback of Physician Performance versus Immediate Reminders to Perform Preventive Care: Effects on Physician Compliance.” Medical Care. 1986;24(8):659–66. [PubMed]
- Tierney WM, McDonald CJ, Hui SL, Martin DK. “Computer Predictions of Abnormal Test Results: Effects on Outpatient Testing.” Journal of the American Medical Association. 1988;259(8):1194–8. [PubMed]
- Tierney WM, McDonald CJ, Martin DK, Hui SL, Rogers MP. “Computerized Display of Past Test Results: Effects on Outpatient Testing.” Annals of Internal Medicine. 1987;107(4):569–74. [PubMed]
- Tierney WM, Miller ME, Hui SL, Mcdonald CJ. “Practice Randomization and Clinical Research: The Indiana Experience.” Medical Care. 1991;29(7 suppl.):JS57–64. [PubMed]
- Tierney WM, Miller ME, McDonald CJ. “The Effect on Test Ordering of Informing Physicians of the Charges for Outpatient Diagnostic Tests.” New England Journal of Medicine. 1990;233(21):1499–504.
- Tierney WM, Miller ME, Overhage JM, McDonald CJ. “Physician Inpatient Order-Writing on Microcomputer Workstations: Effects on Resource Utilization.” Journal of the American Medical Association. 1993;269(3):379–83. [PubMed]
- Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. “Effects of Computerized Guidelines for Managing Heart Disease in Primary Care: A Randomized, Controlled Trial.” Journal of General Internal Medicine. 2003;18(12):967–76. [PubMed]
- Tierney WM. “Improving Clinical Decisions and Outcomes with Information: A Review.” International Journal of Medical Informatics. 2001;62(1):1–9. [PubMed]
- Tunis SR, Hayward RS A, Wilson MC, Rubin HR, Bass EB, Johnston M, Steinberg EP. “Internists' Attitudes about Clinical Practice Guidelines.” Annals of Internal Medicine. 1994;120(11):956–63. [PubMed]
- Webster GD. Final Report of the ABIM Patient Satisfaction Project. Philadelphia: American Board of Internal Medicine; 1988.
- Weinberger M, Murray MD, Marrero DG, Brewer N, Lykens M, Harris LE, Tierney WM. “A Pharmaceutical Care Program for Patients with Reactive Airways Disease.” American Review of Health-System Pharmacy. 2001;58(9):791–6.
- Wyrwich KW, Tierney WM, Wolinsky FD. “Further Evidence Supporting a SEM-Based Criterion for Identifying Meaningful Intra-Individual Changes in Health-Related Quality of Life.” Journal of Clinical Epidemiology. 1999;52(9):861–73. [PubMed]
- Zhou XH, Tu W. “Comparisons of Several Independent Population Means When Their Samples Contain Non-Zero Log-Normal and Possibly Zero Observations.” Biometrics. 1999;55(2):645–51. [PubMed]
|
This article has been 
