NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Committee on Evaluating Clinical Applications of Telemedicine; Field MJ, editor. Telemedicine: A Guide to Assessing Telecommunications in Health Care. Washington (DC): National Academies Press (US); 1996.

Cover of Telemedicine

Telemedicine: A Guide to Assessing Telecommunications in Health Care.

Show details

7Evaluating the Effects of Telemedicine on Quality, Access, and Cost

Does telepyschiatry provide more timely access to appropriate behavioral health services than conventional arrangements for patients in a remote rural community? How does it affect patients' health and well-being compared to the alternatives? How do costs compare? Are patients and clinicians satisfied with the services? Would they want to use them in the future? Why or why not? These are the kinds of questions that clinicians, patients, managers, and policymakers want answered about telemedicine.

This chapter focuses on questions about the quality, accessibility, cost, and acceptability of telemedicine services. Additional questions will, however, be relevant for some organizations, some communities, and some evaluations. For example, because many telemedicine programs also serve educational and administrative purposes, evaluations may reasonably seek to assess results in these areas. The committee's evaluation framework likewise provides for strategic objectives such as strengthening an organization's competitive positive. As described in Chapter 5, the evaluation domains proposed by the federal Joint Working Group on Telemedicine included the "health system interface." Differing in form but not significantly in substance, the committee's framework treats this domain as a set of intermediate technical, clinical, and administrative factors that need to be tracked and understood as part of an evaluation of quality, access, cost, and acceptability outcomes.

Broader community effects may also be considered in an evaluation. Policymakers may, for example, be interested in the effects of telemedicine on the survival of rural health care providers and the implications of such effects for the overall economic health of rural areas, including their ability to attract or maintain business, educational, and other resources (OTA, 1991; Council on Competitiveness, 1994; GAO, 1996). For any specific evaluation, the selection of measures and criteria will depend on the telemedicine application, the alternatives to which it is compared, the target clinical problems and populations, the setting, and similar factors.

Evaluation Criteria And Questions

As defined in Chapter 1, an evaluation criterion is a measure, indicator, standard, or similar basis for describing outcomes or making judgments. Because clinical telemedicine varies so much, the committee broadly interpreted its charge to propose a set of evaluation criteria related to its evaluation framework. Applications differ in the medical problems addressed, the evidence base for decisionmaking, and the diagnostic, therapeutic, and other strategies employed. It would have been far beyond the resources for this project to develop operational measures or standards of care specific to the array of teleradiology, teledermatology, telepsychiatry, home health, emergency care, and other applications described in this report.

Rather, the committee started with the set of basic questions about quality, access, and cost that guide much health services research, particularly in the interrelated fields of clinical evaluation and technology assessment (IOM, 1993b, 1995a). Although patient satisfaction measures may be incorporated into assessments of quality of care, particularly in managed care plans (Cleary and McNeil, 1988; Gold and Wooldridge, 1995), more specific questions about patient and clinician satisfaction and other perceptions are presented separately in this chapter. Questions about health outcomes are largely subsumed in the discussion of quality but also enter into assessments of cost-effectiveness.

Table 7.1 lists the broad categories of questions proposed by the committee. The importance of comparing telemedicine to an alternative is highlighted in each question. The note for the table emphasizes that the research design and analytic strategy will need to take into account and control for such factors as the initial condition of patients. Thus, each question should be read with the phrase " other things being equal" as an implicit preface.

TABLE 7.1. Categories of Evaluation Questions for Comparing Telemedicine to Alternative Health Services.


Categories of Evaluation Questions for Comparing Telemedicine to Alternative Health Services.

The next sections of this chapter provide definitions, discuss key concepts, and present additional questions focusing on different aspects of quality, access, cost, and patient and clinician attitudes. These sections should be read in the context of the overall framework presented in Chapter 6. That is, relevant patient and organizational characteristics should be identified and considered as they might affect results. The level of an evaluation—whether it reflects a patient, corporate, or societal perspective—should also be identified. The fit between the project objectives and results and the evaluation sponsor's purposes or strategic plan also needs to be factored into the plan for analysis and the interpretation of results. The human and policy issues identified in Chapters 3 and 4 likewise warrant attention so that evaluation planning casts a wide net for possible benefits and costs of an application.

Some telemedicine evaluations will focus less on individual patients than on populations, including but not limited to those enrolled in managed care plans. Analyses may consider outcomes for an entire patient population or may concentrate on outcomes for the least healthy or most vulnerable groups in a population (e.g., elderly individuals, migrant workers). For example, a telemedicine application might target a high-risk group to test whether surveillance and early intervention could reduce hospitalization and net costs.

Quality Of Care

The ultimate purpose of any medical care is to maintain or improve health and well-being. Thus, how clinical applications of telemedicine affect the quality of care and its outcomes is a central evaluative question—as it is for any health service.

Definitions and Concepts

As defined in Chapter 1, quality of care is "the degree to which health care services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge" (IOM, 1990c).1 A few points about this definition are worth noting.

First, the definition covers both individuals and populations and both current and potential users of health care. This is consistent with an increasing focus in health services research and health policy on how different clinical interventions, programs, and resources can be deployed to the greatest social advantage. Second, because the evidence base about what works in health care is still modest, the definition acknowledges the relevance of professional knowledge, which includes experience and judgment as well as the results of biomedical and clinical research. Third, as is traditional in the literature on quality of care, the definition encompasses the link between the processes and the outcomes of care (Donabedian, 1966, 1982, 1985), although the emphasis in recent years has been on the latter. Many studies of health care quality also search for structural aspects of quality, for example, characteristics of a health system's personnel or organization that are associated with better health outcomes and that can be incorporated into accreditation or credentialing programs.

Finally, the definition deliberately omits resource constraints on the grounds that judgments of what constitutes excellent, acceptable, or unacceptable quality should be independent of constraints on resources. This does not, however, imply that decisionmakers can or should ignore resources in making decisions about what level of quality is desired and affordable.

In recent years, traditional quality assessment and assurance concepts and strategies in health care have been powerfully reshaped by proponents of continuous quality improvement or total quality management models. These models stress internal responsibility for quality rather than external regulation. As noted in Chapter 6, they also posit planning, control, assessment, and improvement activities grounded in statistical and scientific precepts and driven by data.

Conventionally, three broad types of quality problems have been differentiated. They are overuse of care (e.g., unnecessary telemedicine consultations); underuse of care (e.g., failure to refer a patient for a necessary consultation); and poor technical or interpersonal performance (e.g., incorrect interpretation of pathology specimen or inattention to patient concerns). In principle, no one of these three problems is more important than any other. Depending, however, on the setting, the clinical condition, the predominant financing mechanism, and other circumstances, one area may warrant more attention than another in a particular telemedicine evaluation. For instance, as discussed in Chapter 4, policymakers have been concerned that payment for telemedicine in a fee-for-service context might lead to excessive consultations that might, in turn, lead to overuse of diagnostic or therapeutic services for which the benefit would not be worth the risk. In capitated environments, the worry has been that financial incentives might lead to underuse of appropriate face-to-face consultations or other services and to poorer performance in the interpersonal aspects of patient care, including good communication between clinician and patient.

For purposes of this discussion and consistent with past usage in IOM reports, appropriate care is defined as care for which "the expected health benefit [exceeds] the expected negative consequences by a sufficient margin" that the care is worth providing (Park et al., 1986, p. 6). At what point is the extra margin of expected benefit such that an intervention might be "worth" any additional risk, therefore making the intervention appropriate? Answering this question necessarily involves subjective—and sometimes controversial—judgments as well as objective clinical information. Such judgments may be arrived at through expert consensus processes or by reference to other interventions that have been accepted as standard practice.

The clinical effects of telemedicine applications can be measured and compared at several levels. One may, for example, look for effects on the process of care or for effects on the outcomes of care or both. In a discussion of the impact of diagnostic technologies, Fineberg and colleagues (1977) distinguished several process and outcome dimensions that might appropriately be assessed by evaluators. These dimensions include

  • technical capacity—whether a technology is safe, accurate, and reliable (e.g., how do transmitted digital images compare to films?);
  • diagnostic accuracy—whether a technology contributes to a correct diagnosis (e.g., was an initial dermatology diagnosis by a primary care clinician corrected after review by a dermatologist?);
  • diagnostic impact—whether a technology provides diagnostic information that is useful in making a diagnosis (e.g., after the telemedicine consult, is a face-to-face consultation still necessary?);
  • therapeutic impact—whether a technology influences patient management or therapy (e.g., do paramedics perform better when they have access to emergency cardiac telemetry?); and
  • patient outcome—whether a technology improves patients' health and well-being (e.g., are postsurgical patients telemonitored in a nursing home more or less likely to develop wound infections than patients remaining in the hospital?).

The first four dimensions involve processes of care. The last involves outcomes. Both categories figure in the question set presented below.

In principle, several kinds of process and outcomes measures might be relevant for any specific telemedicine application. For example, in North Carolina, researchers studying an emergency medicine project involving rural emergency departments and four medical schools plan to collect process of care, utilization, and outcomes data on "patient flow, time to diagnosis, effectiveness of specialty consultation, types of cases, appropriateness of intervention at local levels, and patient stabilization” (Evaluation Plan of the North Carolina Emergency Consult Network, p. 2).

Questions about Quality of Care and Patient Outcomes

As explained above, the committee concluded that it would identify basic questions about quality of care to guide evaluators in devising questions and criteria specific to their telemedicine project, its objectives, and its context. Table 7.2 lists these questions. Some measures such as survival appear to have limited relevance for most telemedicine uses, although mortality measures might be considered in evaluating certain applications in emergency care and home monitoring.

TABLE 7.2. Evaluating Quality of Care and Health Outcomes.


Evaluating Quality of Care and Health Outcomes.

Processes of Care

The first set of measures in Table 7.2 relate to processes of care. Process of care measures are useful in their own right as they help evaluators to understand how care is provided, how an intervention changes other aspects of the care process, and how processes of care might be improved to achieve better outcomes or greater efficiency (Donabedian, 1966, 1982; IOM, 1990a; Wilson and Cleary, 1995; Wilson and Kaplan, 1995).

It is important to note that the process measures discussed here do not cover a variety of important but often routine quality assurance procedures. For example, those involved with digital radiology and teleradiology have developed and are still improving quality assurance methods for testing, calibrating, and otherwise monitoring and maintaining equipment at central and remote sites (Forsberg, 1995).

Sometimes, process measures are employed as proxies for health outcomes when data on the latter are limited or unavailable. For example, an early retrospective evaluation of Army telemedicine in Somalia and other sites was able to determine whether the diagnosis or patient care plan changed after the telemedicine consultation, but evaluators lacked data to judge whether the change made a difference in patient outcomes (Walters, forthcoming). Difference in diagnosis may be the most common outcomes-related measure found in tele-medicine evaluations to date. Ideally, previous research should have demonstrated a link between the proxy variable and the desired health outcome. Depending on the objective of an evaluation, the nature of the clinical problem and the intervention, and the resources and data available, the same variable (e.g., vaccination rates) may be treated as an outcome in some studies and as a process measure in others.

Characteristics of a specific telemedicine project may affect the interpretation of utilization and other process information. For example, given similar patient populations, one might expect an experienced primary care physician to refer fewer patients for specialty consultations than a nurse practitioner. One hypothesis for exploration is that the utility of telemedicine is greater when the (initial) difference between the skills and experience of consultant and the referring clinician is greater.

Outcomes of Care

The value of process measures notwithstanding, decisionmakers, clinicians, and patients have increasingly demanded information on outcomes and questioned the assumption that conformance to procedural standards equates to good health outcomes (Relman, 1988; IOM, 1990c; Lansky, 1993). As suggested in Table 7.2, measures of patient outcomes may focus on

  • clinical status (physiological and cognitive);
  • mental and emotional well-being;
  • feelings of energy and vitality; or
  • functional capacity (e.g., ability to perform various tasks related to personal life or employment).

Patient outcomes are generally considered to include not just desired endpoints of health care (e.g., reduced mortality, improved functioning) but also a broad range of immediate and intermediate results (e.g., reduced blood pressure, higher vaccination rates, fewer hospital readmissions for surgical complications) (Brenner et al., 1995). Because patient outcomes data are often difficult to obtain for longer-term outcomes and outcomes that occur outside the hospital, immediate or intermediate clinical results (e.g., physiological signs such as blood pressure or postoperative complications) are frequently used in place of longer-term results. The advantage of such measures is that they may be more directly and strongly linked to elements of a clinical intervention. Their great disadvantage is that their relationship to outcomes of greater relevance to patients (e.g., function) may be theoretical rather than documented through prior research.

The longer the interval that defines an episode of care or a long-term outcome and the more sources of care (and record systems) involved, the more difficult it is to obtain information. Eventually, the integrated, longitudinal computer-based patient record should overcome some of the difficulties in securing satisfactory shorter-and longer-term outcomes data.

A very large literature has accumulated on categories of health outcomes and the tools for measuring them (see, e.g., the quality primer in IOM, 1990c, Vol. II; Lohr, 1992; McDowell and Newall, 1993; CHPS, 1995; Fowler, 1995). Tools for assessing clinical performance and health outcomes have progressed considerably in recent years as methodologists and researchers have tested and improved the validity and reliability of measures and made them more relevant and usable in routine clinical practice. For example, health services researchers have developed shorter and more easily used instruments to measure health status. They also have devised both generic measures and more focused instruments for specific clinical conditions (e.g., diabetes) and settings (e.g., ambulatory care).

Each telemedicine evaluation will have to select quality and outcomes measures that fit the patients, settings, services, desired outcomes, and other characteristics of its project. In some cases, well-established instruments (e.g., for measuring depression or determining patients' assessment of their quality of life) may be available and appropriate for measuring patient outcomes. In other cases, evaluators will have to create measures and data collection instruments, with less confidence in their validity and reliability (see the last section of this chapter).

Adjustments for Patient Risk or Severity of Illness

Proper interpretation of patient outcomes data requires good information on patient characteristics, in particular, their health status. Comparisons of clinical interventions or programs should be adjusted statistically to account for differences in patient risk factors. These adjustments are also essential for proper interpretation of comparisons involving the costs of patient care alternatives.

Various schemes have been devised to measure and adjust for differences in the seriousness of patients' medical status (Thomas and Ashcraft, 1989, 1991; Iezzoni, 1992; Hopkins and Carroll, 1994). Some focus on care settings (e.g., intensive care units) whereas others are more general. Some are designed less for quality assessment purposes than for assuring that capitated, per case, or other payment mechanisms do not pay too much for healthier than average patients and too little for sicker patients. Debate continues on the strengths and limitations of different strategies, but the committee stresses the importance of attempting to identify and adjust for differences in patient status.

Other Quality of Care Issues

As noted elsewhere in this report, primary care physicians or nurse practitioners who participate with patients in telemedicine consultations may learn more about clinical problems that they once referred to specialists and, thereby, become more proficient at identifying and managing repeat problems on their own. Telemedicine may, in this respect, be analogous to the informal "curbside" consultation about a specific patient, a process that clinicians may value more highly than consulting a journal or undertaking formal continuing medical education.

The extent to which clinical applications of telemedicine have this kind of educational effect is not well documented. The committee believes this area warrants further study. Such study should consider not only changes in knowledge but also changes in practice and, preferably, in short- or long-term health outcomes. In addition, systems-oriented evaluations may be warranted to identify how telemedicine systems can support local quality improvement activities through (a) access to data resources, medical literature, and expert opinion, (b) focused educational interventions and mentoring initiatives; and (c) interorganizational collaborations.

Another question related to the impact of telemedicine use on users' knowledge or skills is whether clinicians become more skilled in telemedicine (e.g., relating more effectively to patients during interactive video consultations, reading transmitted images more accurately) as they use a particular application more often. Does some kind of learning curve exist for certain applications? If so, would studies find that a higher volume of use was associated with better outcomes beyond the learning period?2 What might this imply for programs with persistently low numbers of telemedicine consultations? Might some minimum number of cases be suggested as a floor? More generally, what kind of procedures, if any, are appropriate for training and then certifying proficiency in a particular telemedicine application?

How the volume-outcome hypothesis might apply for telemedicine is largely unexplored. One possibility is that quality of care would improve if the consultations involved both high-volume consultants and services those for which high volume was linked to better outcomes. Another possibility is that specialists who had received referrals that were subsequently handled through telemedicine consultation (with a different specialist) might lose the volume of cases needed to maintain their proficiency in diagnosing or treating certain problems. Even if local specialists were reasonably available, would more complex cases be diverted to distant telemedicine consultants? These unanswered questions have implications for both quality of care and access to care. The latter topic is discussed next.

Evaluating Access

From its beginnings, one of the major promises of telemedicine has been that it would improve access to health services for people living in rural or remote areas where medical professionals and facilities were scarce or altogether absent. This promise has been the rationale behind three decades' worth of demonstration projects targeted at rural areas. More recently, the potential for telemedicine to improve access for other groups—for example, the inner-city poor and the urban and suburban homebound—has attracted interest. An emerging issue is how a restructured health care system might employ telemedicine as part of increasingly aggressive strategies to manage patient access to services, especially hospital care and referrals to specialists.

Although the emphasis in telemedicine has been on geography or distance from health care providers as a barrier to timely care, other barriers to access also need to be considered in an evaluation framework (IOM, 1993a,b) A more comprehensive list of barriers would include

  • significant distance from primary, secondary, and tertiary medical services;
  • poor transportation (e.g., lack of an automobile, limited or nonexistent bus service), even for relatively short distances;
  • inadequate financial resources, particularly insurance coverage or directly subsidized services;
  • family, educational, and cultural factors (e.g., illiteracy, distrust of technology);
  • delivery system characteristics, including poor coordination of care, long waiting times for appointments, inadequate numbers or kinds of specialists, and bureaucratic obstacles to services; and
  • gaps in our knowledge about how these factors interact to affect the use of services and what can be done to overcome or eliminate barriers to access.

Further, access involves more than an open door to personal health services provided by health professionals. Today, telecommunications and information technologies permit greater access to health information and thereby allow patients, potential patients, and families to learn more about health problems, care options, and prevention strategies. For those without computers or even telephones, however, access to these information resources is more a promise than a reality. Community clinics may be able to provide some with access to information resources, but funding for such services and for the clinics themselves is vulnerable to retrenchment in public services and budgets. Deficits in literacy and language skills may create further difficulties for disadvantaged populations. The gap in access may actually widen if information services improve only for the more affluent and educated.

The committee notes that the availability of telemedicine for clinical, educational, and other purposes may aid in the recruitment and retention of health professionals in underserved areas, although this has not yet been systematically evaluated. Telemedicine has the potential to tie rural practitioners more closely to experts and colleagues in more urban areas and, thus, to reduce isolation. To the extent that managed care networks reduce professional opportunities in urban and suburban communities and drive physicians and others to consider practice in underserved areas, clinical and educational uses of telemedicine could provide social and intellectual support that would ease such relocations.

Definitions and Concepts

Access was defined in Chapter 1 as the timely receipt of appropriate health care. More informally, access might be described as the availability of the right care at the right time without undue burden. The latter conceptualization maintains the notions of timeliness and appropriateness but adds two elements to the understanding of access: availability and burden.

One element, availability, incorporates the notion of services that stand ready for use if and when needed. Residents of an area may be considered to have access to available services (e.g., a nearby emergency department) even if most people never need or use them. The other element in the informal definition, undue burden, suggests that the difficulty of actually obtaining appropriate services should be considered in evaluating access. For example, if a telepsychiatry consultation saves a patient and others a risky trip over bad winter roads, then it has affected access. Similarly, if telemedicine helps ventilator-dependent patients avoid the burden of transport from the home to a physician's office, then access is affected. What constitutes an undue burden will clearly vary across individuals with differing incomes, insurance coverage, transportation resources, physical limitations, employment situations, and other characteristics. Whether a reduced burden for a patient is worth the cost involved is an important but separate question.

Both formal and informal conceptualizations of access imply that the evaluative focus ought to be on people's ability to get appropriate care rather than on their ability to get any service, whether appropriate or not. Although this point is easy to make, it is more difficult to translate into operational measures, in part because of disagreement about what constitutes appropriate care for specific problems and in part because of the difficulty of data collection or interpretation. As a result, resources (e.g., hospital beds or physicians per 1,000 population) are often used as indicators and may be acceptable for some evaluations. Nonetheless, the use of such measures may erroneously imply to some that more physical resources automatically equate to more health benefit.

One additional distinction may need to be considered. That is, does a telemedicine application affect access only when it directly involves the patient (e.g., as does an interactive video consultation for a psychiatric problem) or does it also affect access when mediated through a clinician (e.g., as in the typical teleradiology consultation)? If telemedicine allows a clinician quicker access to important information that would support a decision to treat locally rather than transfer or refer, then the patient could be said to have more timely access to appropriate care and, thus, better access to care.

Clearly, access as defined here involves multiple dimensions, some of which (e.g., appropriate care) overlap with quality and cost evaluations. Moreover, the committee recognizes that transforming concepts such as "timely," "appropriate," and "undue burden" into operational measures and evaluating results may involve considerable subjective judgment.

Questions about Access to Care

Table 7.3 lists the questions related to access proposed by the committee. Again, the choice, formulation, and interpretation of specific questions will depend on the type of application, the context in which it is employed, the research design, and the resources available for evaluation. Some questions may overlap with those used in evaluating other outcomes, such as patient satisfaction.

TABLE 7.3. Evaluating Access to Care.


Evaluating Access to Care.

In principle, access may be measured at the individual, group, or the population level. Because access questions are often raised in the context of concerns about disadvantaged groups, the policy and evaluation focus is, in fact, often on populations or population subgroups. A 1993 IOM report on indicators of access to health care identified several population-based utilization and outcomes measures that could be employed to monitor national access objectives (often with a focus on identified problem groups such as rural or minority populations). For example, one proposed indicator of the lack of access to timely and appropriate treatment was avoidable hospitalization for chronic diseases. The suggested measures for this indicator included admission rates for selected ambulatory-care-sensitive conditions (e.g., asthma, diabetes) as determined from hospital discharge abstracts for groups defined by income (based on zip code information). Other access indicators included rates of vaccine-preventable childhood diseases and rates of immunizations. For all such indicators and measures, the 1993 report discussed the nature and limits of available data sources.

Telemedicine remains at such an early stage of implementation and diffusion that the committee would not expect it to have had effects that would be evident from such population-based analyses. Furthermore, information on the use of telemedicine services is not routinely available in major national databases so that it would not now be possible to link the availability of telemedicine in different areas to differences in access measures. The kinds of routine and specialized surveys and other data collection instruments used to obtain information for the databases described in the IOM report on access may, however, provide useful models for those devising measurement and data collection strategies for telemedicine projects employed by health systems that serve well-defined populations. Even so, relatively few clinics, health plans, or organizations have the combination of reasonably well-defined patient or enrollee populations, detailed clinical and administrative databases, and resources for special surveys that more sophisticated measures of access would require.

In reviewing telemedicine evaluation activities, the committee identified several access-related indicators that evaluators had used or hoped to obtain through existing or specially created data collection processes. These indicators, which do not—in and of themselves—consider the appropriateness of services, include

  • use of telemedicine services over time;
  • changes in the number of traditional consultations;
  • changes in waiting time for specialist appointments;
  • changes in rates of missed appointments for consultations;
  • patient willingness to participate in a telemedicine consultation; and
  • patient or clinician attitudes about the timeliness of consultations and the burden of different consultation options.

Particularly with the increase in competition in the health care system, health care organizations have established a variety of performance indicators related to certain dimensions of access. These include the wait time for different kinds of services (e.g., urgent versus nonurgent problems), time "on hold" for a telephone call, number of calls lost, and frequency of busy signals. What constitutes acceptable performance appears to vary depending on purchaser and patient expectations, regulations, resources, and other factors.

Evaluating Costs And Cost-Effectiveness Of Telemedicine3

Although improved access to health care has been the motivating force behind many telemedicine applications, reduced health care costs or reduced rates of cost escalation have dominated many other health care initiatives. These include efforts to increase competition in health services, to change methods for paying clinicians and institutions, to make patients more conscious of costs, and to identify and discourage overuse of health services. In this environment, the costs and cost-effectiveness of telemedicine applications compared to conventional health services are understandably central concerns of decisionmakers.

Level and Perspective of the Analysis

As discussed generally in Chapter 6, it is essential to specify the level and perspective of an analysis and to include or exclude costs accordingly. Most relevant for many public policy decisions is the societal perspective, which encompasses the total costs of resources used to provide a service through telemedicine or alternative means. Nonetheless, it may also be appropriate for such analyses to identify how monetary costs and savings are distributed among particular parties. Entities such as insurers, providers, and patients bear variable portions of total costs and reap variable amounts of any cost savings.

Thus, an analysis based on a private insurer's perspective might incorporate costs only for health care benefits or services covered by the insurance plan and exclude any deductibles and copayments or uncovered medical and other expenses (e.g., transportation) borne by the insured and any bad debts absorbed by providers for patients who could not pay their share of costs. Hospitals and physician groups would generate a somewhat different set of included and excluded costs, as would patients. Moreover, in addition to costs for uncovered services and copayments or coinsurance, patients and other members of the population at risk experience health effects—positive and negative.

For health plans or providers paid on a capitation basis, the perspectives of payers and providers may be melded and reshaped as these parties assume financial responsibility for a comprehensive set of benefits for a defined population at risk. As discussed in Chapter 4, the financial incentives of capitation reward providers for delivering care in the most efficient manner. If telemedicine offers efficiencies compared to its alternatives, managed care plans and capitated systems are more likely to realize these benefits and to invest in telemedicine technologies. Further, to the extent that managed care and capitated delivery systems encompass a broader range of services and health professionals and to the extent that they maintain a stable enrollee population over time (which cannot be assumed), they may come closer than traditional insurers and providers to internalizing the total costs of alternative ways of managing medical conditions.

The perspective of analysis may be particularly important in the treatment of transportation costs. Health care organizations, integrated delivery systems, and managed care plans may or may not internalize the travel costs of physicians and other health professionals delivering care to people at a distance. Within traditional fee-for-service payment and private indemnity insurance, it has been unusual for plans to cover transportation of patients, except for ambulances or other special vehicles and for emergencies. However, some public programs, such as Medicaid, have covered more routine patient transportation, even under fee-for-service arrangements. For states, the prospect of reduced transportation costs has been a major attraction of prison telemedicine programs.

Definitions and Concepts

Costs are intended to measure the value of resource use associated with an intervention. The hallmark of economic evaluation is comparison of the costs and benefits of alternative ways of managing a condition. Cost-effectiveness analysis, the most common technique, compares costs and health effects of at least two alternatives. For example, a psychiatric consult or counseling session conducted through telemedicine could be compared to one conducted in person. Cost-effectiveness analysis expresses health effects in natural units, such as years of life gained or cases of cancer prevented.4 By contrast, cost-benefit analysis expresses both costs and benefits (e.g., years of life gained) in monetary terms. The following discussion generally reflects basic principles of cost and cost-effectiveness analysis as identified in a number of sources (see, e.g., Weinstein and Stason, 1977; Warner and Luce, 1982; Drummond et al., 1987; Eisenberg, 1989; Sisk, 1990; Udvarhelyi et al., 1992; Kee, 1994; OTA, 1994).

It is not meaningful to question whether telemedicine per se is a good investment, because its worth—like that of any technology—depends on the circumstances of its use. The meaningful issue for evaluation is whether telemedicine is a good investment for a specific purpose, compared to an alternative(s). Ideally, an evaluation should specify the full range of actual alternatives, so that the results are relevant to the decisions that people face.

To calculate the total cost of telemedicine, one should, in principle, include the costs of all resources to all parties. Cost calculations should also factor in any savings or changes in productivity associated with the application. For example, the potential economic benefits of digital radiology networks include increases in the average number of images read per radiologist per week and reductions in the number of retaken or mislaid images, the times for image location and retrieval, and the physical space required for storage (Vanden and Strauss, 1995). Such benefits may be highly dependent on the technical characteristics and scope of an installation, for example, whether digital imaging is used on an institution-wide rather than supplemental or incremental basis or whether any major infrastructure costs are shared with other applications.

Capital costs for building, major remodeling, or large equipment expenses should be included if the project calls for telemedicine capacity to be established anew or for existing capacity to be significantly expanded. If a telemedicine project is operational, it may be appropriate to include only variable costs, that is, costs that vary with the level of output, such as the number of radiology consults or counseling sessions per month.

Cost analyses examine the differential, incremental, or marginal costs of one alternative compared to another. If the alternatives (e.g., telemedicine, mail, or personal travel for radiology consults) all equally use the same buildings and certain personnel, then the costs of those common resources will not affect comparative costs and need not be calculated for comparative analysis. The analysis should then focus on costs that differ among the alternatives, including personnel, supplies, and personal transportation and time for the radiologist or patient.

For a telemedicine application that requires an infrastructure with sizable fixed costs that cannot be legitimately shared or assigned in part to other users, the application of these principles implies a higher per unit cost. Similarly, during the start-up period of a program, spreading costs over a small number of cases will also result in high per unit costs. Such costs should decline as technological developments reduce infrastructure costs and make telemedicine more convenient for larger numbers of patients. For example, as health care organizations continue to computerize their medical records and as consumers acquire interactive devices for entertainment or personal communication (e.g., two-way cable services, computer access to the Internet), the costs of adding certain telemedicine services in institutions, offices, and homes will be reduced. (A related but distinct issue is that if parts of the telemedicine infrastructure are subject to rapid obsolescence and need replacing or upgrading, then costs may not decline as much.)

If the health effects or cost implications of telemedicine or its alternatives stretch over time, the future stream of health effects and costs should be discounted to their present value. Discounting reflects the idea that people place a higher value on events or benefits in the present than in the future and that funds invested in the present can reap interest over time. It is not an adjustment for inflation.

Though often used as proxies for the cost of services, "billed charges" are list prices that may contain substantial distortions among services, particularly given the discounted, per case, or other payment arrangements that now apply for a substantial portion of health services. Payments, which are based on actual financial transactions, are usually preferable to charges, although in markets characterized by deep discounts to some payers, they too may be a poor proxy for direct measures of costs. Capitated payments or payments for packages of services, such as diagnosis-related groups (DRGs), however, may not vary with changes in resource use and cost. Documenting the actual use and per unit cost of resources to provide a service is clearly the preferable approach, though much more difficult to do (see, e.g., Williams, 1996).

Conceptual Challenges

Cost analyses of telemedicine face certain conceptual challenges that typify new device-based technologies with sizable fixed costs and multiple potential uses. Cost analyses can address these issues and clarify their implications but cannot definitively resolve them.

One difficulty arises from the varied uses to which a telemedicine system may be put. Parts of the system might be used to support emergency medical services, radiology consults, interactive patient counseling sessions, and monitoring of patients in their homes. Although each application may have costs specific to its use, such as certain personnel and supplies, all the applications may share other costs related to certain equipment and perhaps certain personnel and supplies. In contrast to accounting conventions, which apply administrative rules to apportion such joint costs of production, economic principles call for allocating joint costs according to the demand that each service faces (OTA, 1980; Sisk et al., 1991).

Another challenge arises because telemedicine, like other innovations, may lead to expanded indications for use. For example, a telemedicine system may be established to permit more timely diagnosis and treatment of trauma patients in rural areas. Once available and accepted, however, primary care physicians may use telemedicine for less urgent cases that they once handled on their own. Even if per unit costs of telemedicine decline with the greater volume, total use and total expenditures may increase.

A third—and by now familiar—challenge is that technological change may render a static study of benefits, harms, and costs outdated, even before the analysis is completed. The diffusion and evolution of technologies, such as those used in telemedicine, is a dynamic process that calls for ongoing evaluation. As adoption and use proceed, telemedicine users are likely to gain greater experience and proficiency that, in turn, may be reflected in lower costs and better outcomes.

To better inform decisionmakers, the possibility of expanded indications or proficiency-related cost reductions may be modeled in a sensitivity analysis. As described in Chapter 6, if uncertainty surrounds the values of certain variables in the evaluation that are considered key, sensitivity analysis can vary the values over reasonable ranges. The findings will indicate how sensitive the results are to these uncertainties.

Question about Costs and Cost-Effectiveness

Table 7.4 summarizes the questions related to costs proposed by the committee. This summary does not distinguish between major categories of costs (e.g., fixed and marginal, capital and operating). Again, the selection of specific measures will depend on the type of application and the context in which it is employed.

TABLE 7.4. Evaluating Health Care Costs and Cost-Effectiveness.


Evaluating Health Care Costs and Cost-Effectiveness.

Some of the questions in Table 7.4 highlight an important but difficult problem for evaluations of telemedicine and, indeed, evaluations of any new technology. That is, what was the effect of the technology on costs over an episode of acute or chronic illness? An evaluation that cannot link services and costs to such episodes may fail to identify care that prevents the need for later, more expensive care or, alternatively, causes a cascade of additional services. For example, home monitoring via telemedicine might encourage quicker identification and response to problems that might be costly to treat if not caught early. Alternatively, such monitoring might identify more borderline problems and generate more home or office visits (see, e.g., Weinberger et al., 1996). As noted elsewhere in this report, the longer the interval that should be tracked in an evaluation, the more difficult become the problems in collecting and properly attributing relevant data.

Decision Rules for Analyzing Cost-Effectiveness Results

For some patterns of cost-effectiveness results, the findings strongly suggest certain decisions. For example,

  • If an alternative is more costly and performs less well (e.g., produces fewer health benefits), it is undesirable.
  • If an alternative is more costly and performs as well, it is undesirable.
  • If an alternative is less costly and performs better, it should be used.
  • If an alternative is less costly and performs as well, it should be used.

In other cases, cost-effectiveness results are more equivocal and judgments will be more subjective. For example,

  • If an alternative is more costly and performs better, are the benefits gained worth the extra costs?
  • If an alternative is less costly and performs less well, are the savings worth the health benefits foregone?

Some analysts have suggested ranges of costs that are considered reasonable, for example, a year of healthy life gained for less than $100,000 (Laupacis et al., 1992). Technology assessments often compare the cost for the option being evaluated to the cost for a well-established technology. Thus, the cost-effectiveness of population-based screening for prostate cancer might be compared to the cost-effectiveness of screening for cervical cancer. In general, cost-effectiveness analysis can guide, but not dictate, judgments about the reasonableness of costs for the health benefits obtained from different health technologies.

Decisionmakers must also consider budgetary limitations as well as cost-effectiveness. Indeed, it may well be that not all technologies considered to be cost-effective (e.g., that can gain a year of healthy life for less than $100,000) can be afforded, given the number of cases potentially involved and the total budgetary implications of different technologies.

Patient And Clinician Perceptions

The discussion of human factors in Chapter 3 stressed patient and clinician perceptions as they may affect the acceptance and adoption of telemedicine. This chapter has noted patient perceptions as a factor to be considered in evaluating quality, access, or cost-effectiveness. They are also important in their own right to the extent that successful telemedicine applications depend on patient and clinician acceptance.

Attempts to assess patient satisfaction or perceptions of quality derive in part from the consumer movement and quality improvement philosophies that have promoted patient autonomy, informed decisionmaking, and patient-centered care (see, e.g., President's Commission, 1983; Eddy, 1990; IOM, 1990c, 1992a; Kasper et al., 1992; and the sections on human factors and continuous quality improvement in Chapters 3 and 6, respectively). In recent years, increased competition in health care markets has also focused the attention of health plans, facilities, and clinicians on how patients or consumers view the quality, accessibility, or cost of the care they offer (Corrigan and Nielson, 1993; Gold and Wooldridge, 1995; Nelson et al., 1995). Employers and governments who purchase coverage for their employees or beneficiaries also have demanded such information. More generally, this is an era characterized by a steady stream of reports about reduced citizen trust in major social institutions and professions and increasing concern about the effect of managed care and selective contracting on physicians' allegiance to their patients. As a result, some effort may be warranted to assess patient trust in the clinicians and health care organizations involved in a telemedicine application.

Clinician perceptions are less often evaluated than patient perceptions, but efforts to improve the effectiveness or efficiency of care may depend on how satisfied those who provide care are with the conditions of practice (e.g., how convenient a telemedicine consultation is). In the committee's view, those evaluating telemedicine have been fairly sensitive to the clinician perspective. They have recognized that the special demands created by the complex and sometimes unfriendly technical infrastructure of telemedicine may frustrate clinicians, slow the provision of care, and create concerns about professional image. The discussion of human factors in Chapter 3 underscores the importance of considering clinician perspectives and needs.

In several telemedicine evaluations, patient satisfaction data appear to be the only patient-level data collected (ORHP, 1995). The committee considers this evaluative focus far too limiting, although it agrees that evaluators should consider patient—and clinician—views. The efforts by federal agencies to strengthen evaluations of federally funded telemedicine projects (as described in Chapter 5) reflects, in part, a recognition of the limitations of patient satisfaction data. Efforts to standardize questionnaires are also under way, as described in Chapter 5.

Methods and Focus

Attempts to assess patient or clinician perspectives usually involve written questionnaires. Questionnaires are attractive tools because they are relatively inexpensive and convenient to administer and analyze, especially if they can be computer scored. They are also relatively flexible and can be administered on-site, by mail, or by telephone, although the validity and reliability of different forms of administration needs to be considered on a case-by-case basis. Some questionnaires focus on discrete encounters (e.g., an office visit) whereas others focus on institutions or organizations (e.g., hospitals or health plans). For the immediate future, telemedicine evaluations will most likely focus on encounters.

The validity and reliability of various instruments for measuring patient satisfaction have been assessed, but more work remains to be done in general and with respect to specific populations, interventions, settings, and outcomes (Ware et al., 1988; Webster, 1989; Hall et al., 1990; IOM, 1990a; Rubin, 1990; Peterson and Wilson, 1992; Carey and Seibert, 1993; Rubin et al., 1993; Bayley et al., 1995; Gold and Wooldridge, 1995; Stump et al., 1995; Etter et al., 1996). Those who use surveys also have to be sensitive to the methodological problems frequently encountered in many kinds of survey research (e.g., nonresponse rates, accuracy of patient recall, positive response bias).

Telemedicine applications potentially offer an unusual opportunity to explore patient satisfaction data in more depth. Because telemedicine encounters may involve video records, it may be possible to match individual encounters with questionnaires and to assess the encounters qualitatively in light of the survey responses. In addition to providing feedback to clinicians and program administrators, evaluators could explore how such qualitative assessments could provide additional guidance about improving practices that appear associated with negative responses. Video taping and critiquing has become relatively common as a teaching tool for medical students. As is true for feedback strategies in general, evaluators would need to provide for appropriate patient consent and be prepared for clinician reaction to negative evaluations.

Questions about Patient and Clinician Perceptions

Tables 7.5 and 7.6 present general questions that may be asked about patient or clinician perceptions. The questions concerning patient satisfaction with telemedicine reflect the approach taken in the applicable Medical Outcomes Study (MOS) visit-specific questions. This approach has been extensively tested (Rubin et al., 1993; Bayley et al., 1995). Although the selection of specific questions will depend on the purposes of a particular evaluation, the design and administration of questionnaires should follow general principles of questionnaire construction (Rossi et al., 1983; Lessler, 1995).

TABLE 7.5. Evaluating Patient Perceptions.


Evaluating Patient Perceptions.

TABLE 7.6. Evaluating Clinician Perceptions.


Evaluating Clinician Perceptions.

Depending on the objectives of an evaluation, relatively general questions may be adequate. If, however, the objective is to pinpoint problems, then questions may need to be not only more specific but also more quantitative. For example, rather than ask generally about whether clinicians found the application convenient, questions might be asked about how much time the consultation took or about whether the hardware or software was difficult to manipulate and how much time was lost to such problems. In addition, in depth interviews may be useful to develop a fuller understanding of how people perceive the advantages and disadvantages of telemedicine.

The consistency and stability of patient perceptions may warrant particular attention. For example, one unpublished study of telecardiology patients found that patients did not find the experience unpleasant (93 percent), an invasion of privacy (95 percent), or unacceptable for lack of physical contact (88 percent). Nonetheless, only 67 percent said they would use the system for emergency or first visits and only 51 percent wanted to use it for follow-up visits (Mattioli, 1996). In an unpublished follow-up survey a year later (which had a 54 percent response rate), a third of the respondents said they would use the system only in an emergency and a third would go elsewhere if it were their only option.

Desirable Attributes Of Evaluation Criteria

Drawing on the work of several groups considering practical but systematic means of improving clinical practice and health care delivery (IOM, 1990c, 1992a,b; Medical Outcomes Trust, 1995; CPRI, 1996), the study committee identified several desirable characteristics or attributes of evaluation criteria (Table 7.7). These attributes are generic, that is, in principle, they should apply to quality, access, and cost criteria alike and to qualitative as well as quantitative measures. They are also ideal attributes; actual criteria will almost certainly fall short on at least some aspects.

TABLE 7.7. Desirable Attributes of Evaluation Criteria.


Desirable Attributes of Evaluation Criteria.

For several of the attributes (including reliability and validity) and certain kinds of clinical measures, a controlled vocabulary (i.e., a precise, common clinical terminology) is important. The need for a controlled vocabulary arises from a common difficulty in clinical research, clinical practice guidelines, and medical informatics: the lack of unambiguous, uniform descriptors of patient problems (see IOM, 1990c, 1992a; Gibson and Middleton, 1994; Ozbolt et al., 1994). For example, terms like "moderate bleeding" or "persistent bleeding" may be interpreted differently in practice by different observers. Bleeding defined in terms of volume loss or hematocrit drops is more precise. Even if definitions are unambiguous, a problem remains if they are not uniformly used. In this context, a controlled vocabulary is one specified by those responsible for an information system and one that precludes users from adding unauthorized terms.

Developing a controlled vocabulary and implementing it are long-term challenges. Several schemes have been developed to increase uniformity in the coding of patient history and physical results, medical diagnoses, or procedures. They go under a variety of abbreviations and acronyms (e.g., ICD-9-CM, CPT-4, SNOMEDIII) and are described in detail elsewhere (e.g., PPRC, 1988; IOM, 1991; AMA, 1993; CAP, 1993; Gibson and Middleton, 1994). To build on these efforts, the National Library of Medicine has developed a Uniform Medical Language System (UMLS) Metathesaurus to map terms used by such schemes.


This chapter has reviewed issues in measuring and evaluating critical outcomes for telemedicine and proposed general evaluation questions in four key areas: quality, access, cost, and patient and clinician perceptions and satisfaction. Depending on the application and clinical problem, the setting and patient population, the objectives of the program, and other factors, evaluations will differ in the outcomes of greatest interest and relevance. As stressed in Chapter 6, the earlier and more precisely evaluation objectives and questions are identified, the more likely it is that the program to be evaluated can be designed and implemented in ways that will help provide useful and credible answers.

Although the questions about quality, access, cost, and patient and clinician perceptions are presented sequentially above, their interrelationships also warrant attention. For example, the timeliness of care—an element of access as defined here—may have important consequences for quality through earlier detection and better management of clinical problems. Similarly, economic analyses of telemedicine do not simply examine costs but attempt to relate the costs of an application to its benefits and to suggest bases for judging whether the benefits are worth the costs in comparison to other alternatives. Judgments are typically based on a balancing of objectives that is contingent on a given evaluation's mix of effects on quality, access, and cost. For evaluations that are beyond the "test of concept" or formative phase, a central question will often be: What do the quality, access, cost, and other results suggest about whether and how the telemedicine program can be sustained beyond the evaluation stage?



The discussion in this section draws on the Institute of Medicine's work over the past decade on quality of care, effectiveness research, and related topics (in addition to IOM, 1990c, see IOM, 1985c, see IOM, 1990a, 1991a, 1992a).


Interest in the link between volume and quality of care has arisen primarily in the context of selected surgical and other procedures. Evidence suggests that surgeons who routinely perform a large number of certain relatively complex procedures tend to have better outcomes than those performing such procedures only occasionally (Flood et al., 1984; Hughes et al., 1987; Luft et al., 1987; Hannan et al., 1989; Woods et al., 1992; Hannan et al., 1992). Some health plans attempt to concentrate patients needing a complex procedure in a few "centers of excellence" that perform the procedure frequently, present evidence of good outcomes, and offer an attractive price.


This section is based in part on a paper drafted by committee members Jane Sisk and Jay Sanders.


Some analysts use the term cost-utility analysis when outcomes are expressed in units (e.g., quality-adjusted life years or QALYS) that are intended to apply commonly across different problems (OTA, 1994).

Copyright © 1996, National Academy of Sciences.
Bookshelf ID: NBK45438


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...