Logo of jgimedspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
J Gen Intern Med. 1998 Jul; 13(7): 469–475.
PMCID: PMC1496989

Influenza Immunization in a Managed Care Organization

Ann M Baker, MPH,1 Bruce McCarthy, MD, MPH,1,4 Virginia F Gurley, MD, MPH,2 and Marianne Ulcickas Yood, MPH1,3



To compare the effects of different types of computer-generated, mailed reminders on the rate of influenza immunization and to analyze the relative cost-effectiveness of the reminders.


Randomized controlled trial.


Multispeciality group practice.

PATIENTS: We studied 24,743 high-risk adult patients aligned with a primary care physician.


Patients were randomized to one of four interventions: (1) no reminder, which served as control; (2) a generic postcard; (3) a personalized postcard from their physician; and (4) a personalized letter from their physician, tailored to their health risk.


The immunization rate was measured using billing data. A telephone survey was conducted in a subgroup of patients to measure reactions to the mailed reminders. To evaluate the cost-effectiveness, a model was constructed that integrated the observed effect of the interventions with published data on the effect of immunization on future inpatient health care costs.


All three of the reminders studied increased the influenza vaccination rate when compared with the control group. The vaccination rate was 40.6% in the control group, 43.5% in the generic postcard group, 44.7% in the personalized postcard group, and 45.2% in the tailored letter group. The rates of immunization increased as the intensity of the intervention increased (p < .0001). Seventy-eight percent of patients in the letter group deemed the intervention useful, and 86% reported that they would like to get reminders in the future. The cost-effectiveness analysis estimated that in a nonepidemic year, the net savings per 100 reminders sent would be $659 for the personalized postcard intervention and $735 for the tailored letter intervention. When these net cost-savings rates were each applied to the entire high-risk cohort of 24,743 patients, the estimated total net savings was $162,940 for the postcard and $181,858 for the tailored letter.


Although the absolute increase in immunization rates with the use of reminders appeared small, the increases translated into substantial cost savings when applied to a large high-risk population. Personalized reminders were somewhat more effective in increasing immunization, and personalized letters tailored to the patients' condition were deemed useful and important by the individuals who received them and had a beneficial indirect effect on patient satisfaction.

Keywords: immunization, influenza vaccination, mailed reminders, cost-effectiveness

Annual influenza vaccination is recommended for all persons aged 65 years and older and for persons with certain chronic medical disorders.1 Influenza is responsible for significant morbidity and decreased productivity during epidemics. Twenty thousand or more excess deaths have been reported during each of 10 different epidemics from 1972–1973 and 1990–1991. More than 90% of the deaths attributed to pneumonia and influenza in these epidemics occurred among persons aged 65 years and older. Influenza has been estimated to cause a yearly average of 4 million excess respiratory illnesses and 16 to 17 million excess bed and restricted activity days in persons over 20 years of age.1

The Centers for Disease Control and Prevention strongly recommend that outpatient clinics and physicians' offices identify patients who should receive the vaccine and use telephone or mailed reminders to encourage vaccination among patients who do not have regularly scheduled visits.2 Computer-based systems have the potential to identify high-risk patients and produce automated mailed reminders recommending vaccination. Some studies,3–9 but not all,10–12 have demonstrated increases in influenza vaccination rates among high-risk patients who were mailed reminders. Shea et al. performed a meta-analysis of published randomized, controlled trials and concluded that computer reminders increased vaccinations when compared with a control group.13 Although this meta-analysis helps to confirm that mailed reminders do improve vaccination rates, the inconsistency of these results suggest that there may be other factors to consider in developing an effective reminder system. Ornstein et al. conducted focus groups with patients to determine barriers to adherence to reminder letters about preventive services and concluded that careful attention to the format and content of reminder letters is necessary to improve adherence to their recommendations. Focus group participants stressed the importance of distinguishing the reminder from a bill, conveying a personally relevant message, and addressing logistic barriers to preventive services.14 Most studies of mailed reminders have focused on their overall effectiveness, while only a few have measured the impact of content and format variation.

Larson et al. found that the vaccination rate varied from 25% to 51% depending on the content of the reminder and concluded that a reminder that emphasized the elements of the Health Belief Model was more effective than no postcard or a postcard with a neutral message.5 Computer-generated tailored messages have been shown to affect several preventive interventions.15, 16 These studies developed sophisticated tailored messages using information that was collected during baseline interviews thought to be relevant to the behavior of interest. Collection of these detailed data in large populations would be impractical, especially on a yearly basis. However, certain pieces of information, such as the patient's risk status for influenza, are available in most health system databases. Physician advice has been shown to be one of the strongest predictors of vaccination,1719 and it is logical to assume that sending a reminder from the patient's personal physician might enhance vaccination rates more than a generic reminder. Two studies tested this hypothesis with conflicting results. Larson et al. found that a personalized postcard from the physician improved influenza vaccination rates,5 but Taplin et al. found that mailing the recommendation from the woman's primary care physician rather than from the program director did not increase the likelihood that the patient would get a mammogram.20

We hypothesized that both personalizing and tailoring the reminder using information easily accessible to us would enhance its effectiveness. This question was studied in a randomized, controlled trial of computer-generated, mailed reminders used to increase influenza vaccination rates in high-risk, adult patients in a large group practice. Specifically, this study attempted to answer three questions: (1) Will a reminder postcard from the patient's personal physician that is personally addressed to the patient be more effective than a generic postcard?, (2) Will a letter from the patient's personal physician that is personally addressed to the patient and contains a content message tailored to the patient's risk factor for influenza be more effective than either postcard strategies?, and (3) Are these reminder interventions cost-effective?


Study Population

This study was conducted within the Henry Ford Medical Group (HFMG), a multispeciality group practice of 1,100 physicians in southeastern Michigan affiliated with the Henry Ford Health System (HFHS). This medical group provides care to patients enrolled in the health system's affiliated nonprofit, mixed-model HMO, Health Alliance Plan, as well as to individuals with fee-for-service financing arrangements. Beginning in 1993, all patients who visited a physician within the HFMG were asked to confirm or choose one primary care physician. This information was recorded in the HFMG computerized appointment scheduling system and classified the patient as “aligned” with the selected physician. The primary care physician alignment process was conducted for all patients, regardless of insurance status. Approximately 57% of the aligned population were HMO members, and the remainder were fee-for-service patients. In addition to the computerized appointment scheduling system, the HFMG maintains computerized demographic information on all patients who have ever visited HFHS as well as computerized billing data for all encounters within HFMG.

The study population included all adult patients aligned with a primary care physician who were at high risk of influenza complications. High-risk criteria included age 65 or older and/or a diagnosis of asthma, diabetes, end-stage renal disease, sickle cell disease, ischemic cardiomyopathy, and nephrotic syndrome. Using the computerized billing data, we identified all patients aligned with an HFMG primary care physician who had an encounter occurring in 1994 or 1995 with an ICD-9 code of asthma (493.XX), end-stage renal disease (585.XX), nephrotic syndrome (581.9), diabetes (250.XX), sickle-cell disease (282.60), or ischemic cardiomyopathy (414.8). The patient's date of birth, gender, race, and marital status were retrieved from the computerized demographic information.

Patients aligned with a primary care physician who were aged 65 years and older as of January 1, 1995, and/or who were billed for any of the above diagnoses during 1994 or 1995, were included in the study and randomized into one of four groups: group 1 was the control group, which received no mailed intervention; group 2 received a generic postcard; group 3 received a personalized postcard from their primary care physician; and group 4 received a personalized tailored letter from their primary care physician.


The automated reminder initiative was a component of a comprehensive influenza immunization program. The program included walk-in influenza clinics during the month of October at all HFMG outpatient clinic locations; posters and take-home postcards in the clinic entrances and waiting areas; and a toll-free information telephone line. A program logo and theme were developed and incorporated into all print media. The program motto was, “Are You Sick of Getting the Flu? … Get Your Flu Shot.” The standard message of the printed materials was based on the Health Belief Model,21 and included a description of who is at risk of contracting influenza, a statement of the fact that influenza can be serious, and assurance that the vaccine is safe and effective. The printed materials also advised individuals to get the influenza vaccine, and listed the influenza clinic locations and operating hours.

The reminders were computer-generated and mailed to the three intervention groups during the third week of September 1995, with the goal of being delivered to the home during the last week of September. The generic postcard included only the standard content message. The personalized postcard was from the primary care physician, addressed to the patient at risk, and contained the standard message. The personalized tailored letter was from the primary care physician, addressed to the patient at risk, and contained a message tailored to the patient's risk factors for influenza (Appendix A). To assess patient reaction to the reminders, patients were randomly selected for a telephone survey until 50 patients per intervention group had been reached by telephone.

Data Analysis

Effectiveness of the Intervention.

To assess the effectiveness of the interventions, the rate of immunization in each intervention group was calculated using billing data. The cohort was then stratified by risk factor (age 65 and older only, age 65 and older plus chronic disorder, age less than 65 plus chronic disorder) and by intervention group. The influenza vaccination rates and 95% confidence intervals around the difference in the rates of vaccination between the intervention and control groups were calculated. To test for linear trend, the interventions were coded as ordinal variables from 1 (control) through 4 (personalized tailored letter) and entered into a linear regression model. When we controlled for potential confounders (age, gender, race, and marital status), there was no material difference between the crude and adjusted estimates; therefore, crude results are presented.


To evaluate the cost-effectiveness of the intervention, a simple model was constructed that integrated the observed data on immunization rates and program costs with published data on the relation between immunization status and subsequent influenza-associated hospitalization costs. For each strategy, the total estimated health care expenditures for the vaccination program and flu-related hospitalization expenses were calculated. To estimate the cumulative impact for a health system the size of HFMG, the model assumes that each intervention was applied to the entire high-risk population. Each intervention strategy was compared with usual care, first assuming a nonepidemic outbreak of influenza would occur, and then repeated assuming an epidemic outbreak. The perspective assumed was that of a managed care organization responsible for health care services.

The observed data used in the model included the proportion of patients in each group that received an influenza vaccine, the cost of vaccination, and the cost of the reminder intervention received. The cost of vaccination included the acquisition cost of the vaccine and immunization supplies as well as the cost of additional nursing time to administer the program. In addition, the model included an estimated cost of treatment for adverse reactions based on the literature.22 The cost of each reminder intervention included the cost of paper, printing, and postage for the reminder and all posters and flyers that were produced to promote the program. These costs were in 1995 dollars. The one-time costs for computer programming and logo design were not included in the model.

The published data used in the model came from Nichols et al., who reported adjusted mean per-person hospitalization costs for influenza and pneumonia, acute and chronic respiratory conditions, and congestive heart failure for elderly noninstitutionalized people who had and had not received influenza immunization in epidemic and nonepidemic years.23 These hospitalization costs were in 1991–1992 dollars; therefore, they were inflated to 1995 dollars using the average annual consumer price index for medical care.24

Net cost savings per 100 reminders sent was calculated for each strategy compared with the control, or usual-care, strategy. These net cost-savings rates were also applied to the entire high-risk cohort of 24,743 patients in our health system to yield estimates of the cumulative savings that would have been realized if each intervention alone had been used with the entire population.


Effectiveness of the Intervention

A total of 24,743 eligible patients were included in the cohort and randomized into the four groups: the control group who did not receive any intervention (n= 6,171), the generic postcard group (n= 6,169), the personalized postcard group (n= 6,252), and the personalized tailored letter group (n= 6,151). The demographic composition of the cohort is summarized in Table 1). There were no material differences in age, gender, race, and marital status in the four groups. Among the entire cohort, 43.5% were vaccinated for influenza: 40.6% in the control, 43.5% in the generic postcard, 44.7% in the personalized postcard, and 45.2% in the tailored letter group. A test for trend showed that the rate of immunization increased as the intensity of the intervention increased (p < .0001) (i.e., the reminder postcard from the patient's primary care physician was more effective than the generic postcard and the personalized tailored letter was more effective than either postcard intervention). This trend persisted among the patients aged 65 years and older and among patients less than age 65 with a chronic condition (Table 2

Table 1
Demographic Characteristics of the Study Population
Table 2
Percentage of Patients Billed for Influenza Vaccine and Percentage Difference from Control by Intervention and Risk Factor

The results of the telephone survey that was conducted to measure patients' reactions to the reminders are summarized in Table 3. The survey responses for the two postcard groups are presented in aggregate because stratification by specific postcard group was not possible. Overall, 64% of the letter group and 39% of the combined postcard group recalled receiving the mailed reminder. In both groups, the majority of those who recalled receiving the reminder also reported reading it. All patients responded that the reminders were easy to read, and the majority found the information to be useful. Ninety-four percent of the letter group and 97% of the postcard group stated that there was nothing that they disliked about the reminder. Eighty-six percent of the letter group and 59% of the postcard group responded that they would like their doctor to send them other mailed reminders to have tests or immunizations in the future.

Table 3
Telephone Survey Results—Patient Reactions to Mailed Reminders


The marginal cost of vaccination was $4.09 per person. Data were not available on the incidence or cost of medical care for vaccine-related events; therefore, the $4.09 vaccination cost includes an estimate from the literature in the amount of $0.69 per vaccination for vaccine-related events.22 The average cost per person for both postcard reminders was $0.34, and the cost of the tailored letter reminder was $0.42. The inflated mean nonepidemic-year hospitalization costs were $268 and $442 for persons aged 65 years and older who were and were not vaccinated, respectively. Similarly, the inflated mean epidemic-year hospitalization costs for person aged 65 years and over were $333 and $682 for those who were and were not vaccinated, respectively.

The cost-effectiveness of the generic postcard intervention was not modeled because it cost the same as the personalized postcard but resulted in a lower immunization rate. Therefore, only the total costs for the personalized postcard intervention and tailored letter intervention were estimated for each nonepidemic-year and epidemic-year model. The results of the models for a nonepidemic year and an epidemic year are presented in Tables 4 and and5,5, respectively. As the tables show, even in a nonepidemic year the savings per 100 reminders was $659 for the postcard and $735 for the letter, and the cumulative savings would be substantial in a population of this size.

Table 4
Projected Cost Savings from an Influenza Vaccination Reminder Program in a Nonepidemic Year
Table 5
Projected Cost Savings from an Influenza Vaccination Reminder Program in an Epidemic Year


This controlled trial attempted to answer three questions: (1) whether a reminder postcard from the patient's primary care physician, personally addressed to the patient, would be more effective than a generic postcard; (2) whether a letter from the patient's primary care physician, personally addressed to the patient and containing a content message tailored to the patient's risk factor for influenza, would be more effective than either postcard strategies; and (3) whether there are differences in the cost-effectiveness of the reminder interventions. The results of this study show that the rates of vaccination and the net savings rate increased with the intensity of the intervention. The reminder postcard from the patient's primary care physician was more effective than the generic postcard. The personalized, tailored letter was more effective than either postcard intervention and had the highest net savings rate. The patients reacted favorably to all of the mailed reminders; however, the letter group recalled receiving the reminders more often and were more likely to want to receive reminders in the future.

The results of this study concur with those of Larson et al., who found that a personalized postcard from the physician improved influenza vaccination rates,5 and with other studies that have shown that physician advice is an important predictor of vaccination.1719

Furthermore, although we were only able to tailor one variable in the tailored letter reminder, “risk for influenza,” the tailored letter was somewhat more effective than the personalized postcard, and it appeared to enhance patient satisfaction. This observation supports the work of others,15, 16 as well as the notion that minimal tailoring can have an impact on the effectiveness of reminders.

It has been suggested that mailed reminders may have a “ceiling effect,” yielding vaccination rates no higher than 55% to 65%.10 This may explain the small increase in the vaccination rates in this study. However, we suspect that this intervention produced an effect greater than we were able to measure because the intervention groups probably also received more vaccinations at non-HFMG clinics, some of which offered free vaccinations. Some may argue that reminders are not cost-effective in highly immunized populations. This study showed that although the impact was small, the intervention still resulted in substantial cost savings when applied to a large high-risk population.

Conversely, this intervention focused on patient reminders as a mechanism for increasing flu vaccination rates. Combining patient reminders with other approaches such as reminders to physicians, which has also been shown to increase vaccination rates,25 may further enhance vaccination rates. Barton and Schoenhaum concluded that each element of their program, patient reminders, physician reminders, and performance feedback, contributed to the overall increase in vaccination rates in an HMO setting.8

Although reminders have clearly been shown to be effective, some physicians question sending mailed reminders for ethical and practical reasons. A study by Kottke et al. reported that more than half of respondents did not care about being reminded or did not want their physicians to remind them when they were due for a Pap test or mammogram.26 Overall, the patients surveyed in our study were satisfied with the reminders and would like to receive reminders in the future.

There are a number of limitations in this study. The published hospitalization costs used in the cost-effectiveness model were derived from utilization data for people aged 65 years and older; therefore, we do not have estimates of the benefit to patients under age 65. The cost-effectiveness model used in our study did not include costs associated with outpatient medical services and, therefore, may underestimate the cost savings from the reminder program. Conversely, this cost-savings model assumed good antigenic match between the vaccine and the dominant circulating influenza strains. Net cost savings would not be likely in any year when the vaccine and dominant virus strains were poorly matched.

At first glance, the impact of the intervention on vaccination rates may appear small, and the differences in the impact between the interventions insignificant; however, the increase translated into substantial cost savings. These results indicate that all of the mailed reminders studied were cost-saving, independent of age, gender, race, or marital status, with the tailored letter providing the largest cost savings. Personalized reminders were somewhat more effective in increasing immunization, and personalized tailored letters may have a beneficial indirect effect on patient satisfaction.


The authors thank Margette Winters, BSN, MBA, Flavia Scarsella, BSN, Richard E. Ward, MD, MBA, and Jennifer Elston Lafata, PhD.

Appendix A

Example of Patient Letter


[merge field - patient name, address]

Dear Mr/Ms. [merge field - patient last name]:

The flu and pneumonia seasons are here. The flu or pneumonia can strike anyone but it is more serious in people who [merge field - criteria options. i.e. are over age 65, have kidney/lung disease, etc]. Your flu shot needs to be given every year, but your pneumonia shot needs to be given only once. Receiving these shots cannot cause you to get the flu or pneumonia. In fact, most people do not have any side effects from these shots at all.

Henry Ford Medical Centers have the flu and pneumonia vaccines available. I urge you to come in for a flu vaccine. If you have never had a pneumonia vaccine, I urge you to receive this shot too. You don't have to make an appointment. Just walk into any of the Henry Ford Medical Centers on the dates and times listed on the enclosed flyer. If you already have a scheduled appointment with me in the fall, you can get your shots at that visit.

The costs of the shots are covered by Medicare and Health Alliance Plan. If you have another insurance or have questions about coverage, please contact your insurance company.

Our Center for Clinical Effectiveness used a computer to identify patients who would benefit from getting the flu vaccine. If you receive this letter in error, please call [department telephone number] so that we can update our records.


Dr. [merge field - primary care physician]


1. US Preventive Services Task Force. Adult immunizations in guide to clinical preventive services. Guide to Clinical Preventive Services. 2nd ed. Baltimore, Md: Williams & Wilkins; 1996. pp. 791–814. In:
2. Prevention and control of influenza, part I: vaccines. Recommendations of the Advisory Committee of Immunization Practices (ACIP) MMWR. 1993;42:1–14. [PubMed]
3. Brimberry R. Vaccination of high-risk patients for influenza, a comparison of telephone and mail reminder methods. J Fam Pract. 1988;26(4):397–400. [PubMed]
4. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. Can Med Assoc J. 1986;135(9):991–7. [PMC free article] [PubMed]
5. Larson EB, Bergman J, Heidrich F, Alvin BL, Schneeweiss R. Do postcard reminders improve influenza vaccination compliance? A prospective trial of different postcard “cues. ” Med Care. 1982;20(6):639–48. [PubMed]
6. Mullooly JP. Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting. Am J Public Health. 1987;77(5):626–7. [PMC free article] [PubMed]
7. Spaulding SA, Kugler JP. Influenza immunization: the impact of notifying patients of high-risk status. J Fam Pract. 1991;33(5):485–98. [PubMed]
8. Barton MB, Schoenhaum SC. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback. Am J Public Health. 1990;80(5):534–6. [PMC free article] [PubMed]
9. Szilagyi PG, Rodewald LE, Savageau J, Yoos L, Doane C. Improving influenza rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination compliance. Pediatrics. 1992;90(6):871–5. [PubMed]
10. Buchner DM, Larson EB, White RF. Influenza vaccination in community elderly: a controlled trial of postcard reminders. J Am Geriatr Soc. 1987;35(8):755–60. [PubMed]
11. Buffington J, Bell KM, LaForce FM. A target-based model for increasing influenza immunizations in private practice. J Gen Intern Med. 1991;6(3):204–9. [PubMed]
12. Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial. J Gen Intern Med. 1992;7(5):535–7. [PubMed]
13. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inf Assoc. 1996;3(6):399–408. [PMC free article] [PubMed]
14. Ornstein SM, Musham C, Redi A, Jenkins RG, Zempl D, Garr DR. Barriers to adherence to preventive services reminder letters: the patient's perspective. J Fam Pract. 1993;36(2):195–200. [PubMed]
15. Skinner CS, Strecher VJ, Hospers H. Physicians' recommendations for mammography: do tailored messages make a difference. Am J Public Health. 1994;84(1):43–9. [PMC free article] [PubMed]
16. Strecher VJ, Kreuter M, Den Boer DD, Kobrin S, Hospers HJ, Skinner CS. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Pract. 1994;39(3):262–70. [PubMed]
17. Nichol KL, Lofgren PR, Gapinski J. Influenza vaccination. Knowledge, attitudes, and behavior among high-risk outpatients. Arch Intern Med. 1992;152:106–10. [PubMed]
18. Frank JW, Henderson M, McMurray L. Influenza vaccination in the elderly, 1: determinants of acceptance. Can Med Assoc J. 1985;132:371–5. [PMC free article] [PubMed]
19. Adult immunizations: knowledge, attitudes and practices. DeKalb and Fulton Counties, Georgia. MMWR. 1988;37:657–61. [PubMed]
20. Taplin SH, Anderman C, Grothaus L, Curry S, Montano D. Using physician correspondence and postcard reminders to promote mammography use. Am J Public Health. 1994;84(4):571–4. [PMC free article] [PubMed]
21. Janz NK, Becker MH. The health belief model: a decade later. Health Educ Q. 1984;11(1):1–47. [PubMed]
22. Nichol KL, Lind A, Margolis KL, et al. The effectiveness of vaccination against influenza in healthy, working adults. N Engl J Med. 1995;333(14):889–93. [PubMed]
23. Nichol KL, Margolis KL, Wuorenma J, von Sternberg T. The efficacy and cost effectiveness of vaccination against influenza among elderly persons living in the community. N Engl J Med. 1994;331(12):778–84. [PubMed]
24. National Center for Health Statistics . Health, United States, 1995. Hyattsville, Md: Public Health Service; 1996.
25. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccinations on morbidity during influenza epidemics. MD Comput. 1992;9:304–12. [PubMed]
26. Kottke TE, Trapp MA, Fores MM, et al. Cancer screening behaviors and attitudes of women in southeastern Minnesota. JAMA. 1995;273(14):1099–105. [PubMed]

Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine
PubReader format: click here to try


Save items

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • Cited in Books
    Cited in Books
    NCBI Bookshelf books that cite the current articles.
  • MedGen
    Related information in MedGen
  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...