U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Solaiman B, Cohen IG, editors. Research Handbook on Health, AI and the Law. Cheltenham, UK: Edward Elgar Publishing Ltd; 2024 Jul 16. doi: 10.4337/9781802205657.ch09

Cover of Research Handbook on Health, AI and the Law

Research Handbook on Health, AI and the Law.

Show details

Chapter 9Liability for use of artificial intelligence in medicine1

, , and .

While artificial intelligence has substantial potential to improve medical practice, errors will certainly occur, sometimes resulting in injury. Who will be liable? Questions of liability for AI-related injury raise not only immediate concerns for potentially liable parties, but also broader systemic questions about how AI will be developed and adopted. The landscape of liability is complex, involving healthcare providers and institutions and the developers of AI systems. The landscape is also evolving, as the technology remains new and explicit legal consideration is relatively scarce. In this chapter, we consider these three principal loci of liability: individual healthcare providers, focused on physicians; institutions, focused on hospitals; and developers. We describe current doctrine, make tentative predictions and outline potential systemic implications.

1. Introduction

While artificial intelligence (AI) has substantial potential to improve medical practice, errors will certainly occur, sometimes resulting in injury. Who will be liable? Questions of liability for AI-related injury raise not only immediate concerns for potentially liable parties but also broader systemic questions about how AI will be developed and adopted. The landscape of liability is complex, involving healthcare providers and institutions and the developers of AI systems. In this chapter, we consider these three principal loci of liability. At the outset, we note a few issues that shape our analysis.

First, the field of tort liability for AI is still evolving. As of this writing, healthcare AI liability has still not been directly addressed in court cases, mostly because the technology itself is so new and is still being implemented. Accordingly, we consider general principles of tort law and how they are most likely to apply.

Second, causation will often be challenging in AI tort contexts. Demonstrating the cause of an injury is already often hard in the medical context, where outcomes are frequently probabilistic rather than deterministic. Adding in AI models that are often nonintuitive and sometimes inscrutable will likely make causation even more challenging to demonstrate.

Third, we focus on a United States perspective. The principles we discuss are generalisable at some level, but ultimately, there is enough complexity that accurately capturing international differences is infeasible in the space available. We do note in the conclusion some potentially substantial changes on the European horizon.

Fourth, from a systemic perspective, individual healthcare professional liability, though complex, represents only one piece of a larger puzzle that system designers must try to put together to achieve a comprehensive and optimally designed liability system. Many players interact in the medical AI space, including actors who might carry liability and regulators who may shape it. First, AI developers will make many key choices, at least partially guided by the liability system, regarding the underlying AI – will it be locked or adaptive? Will the choice architecture make it easy or difficult to ‘overrule’ the system? What data set will the system use? Second, the US Food and Drug Administration (FDA) will (sometimes) determine the scope of premarket review, if any, for medical AI, and depending on how those requirements are set up, it may or may not preempt some tort liability. Third, AI may be acquired by a hospital system, which may also have co-developed it, trained the AI on the hospital’s own Electronic Health Record (EHR) data, or even developed it entirely in-house. In part, liability systems will guide decisions on what AI to purchase, how to test it and how to integrate the AI system into nurse and physician workflow. A hospital system must decide how to invite or require physicians, nurses and other healthcare providers to use the system – will it adopt measures to try to nudge towards use or even, if legally possible, require consultation with the system as part of the standard of care? Fourth, healthcare providers will actually use AI. Physicians have historically tended to be independent contractors (less so recently, and especially since the onset of the COVID-19 pandemic),2 while nurses tend to be employees, such that the law may treat them differently, as we discuss below. To the extent that they have discretion, healthcare workers will need to decide whether to use AI when offered and when to follow versus ignore an AI-based recommendation – and may face liability for those decisions. Fifth, health insurers and other payers must decide whether to reimburse the hospital for its use/purchase of the AI itself. They also have to decide when to reimburse for a service in relation to what the AI recommends or fails to recommend – that is, can they refuse to reimburse the full costs of a more expensive service when the AI recommends a less expensive one? These decisions may carry liability consequences. Sixth and finally, medical malpractice insurers must decide whether they will cover and how they will defend physicians who follow or fail to follow AI recommendations and get sued.

If – as many think it ought to be – tort liability is an important way to guide behaviour in setting the rules for one level of this problem, one must consider how it will interact with the rules at all the other levels, or else one can create bad incentives even if things go ‘right’ at a particular level. The interactions involved in medical AI create a daunting landscape for such questions.

But before we can even begin to undertake that monumental task, we have to at least understand each individual level on its own. We consider three important potential loci of liability: individual healthcare providers, focused on physicians; institutions, focused on hospitals; and developers.

2. Physician Liability

The use of AI systems in medicine raises unsettled questions about the liability of healthcare providers such as physicians, nurses and other practitioners when patient injury results from problems with AI.3 While we use physicians as our example here, similar patterns apply to other providers who owe duties of care to their patients.

At base, physicians have a duty to treat their patients according to the standard of care. While different states express the standard of care differently, it is typically something like the care that would be provided by a competent physician of the same speciality, taking into consideration the resources that are available. Most states apply a national standard of care, but some are more solicitous of local practice. The interactions between AI and the standard of care are complex and are likely to change over time. Since no cases have yet squarely addressed how the standard of care is altered by the use of an AI system, our analysis relies on the application of medical malpractice law more generally.4

To explore how malpractice law shapes the potential liability of physicians using AI in different ways, we consider a stylised fact pattern, adapted from prior work, where AI makes a recommendation either according to the standard of care or not, the recommendation is correct or not, and the physician follows it or not.5 The fact pattern is highly stylised to make points clearer; in the real world, these decisions are shrouded in probabilities and more complex points which we explore below. Moreover, we use very simplified binary language such as healed/injured and right choice/wrong choice just for illustrative purposes.

Assume a physician is treating a new patient with chronic migraines. The standard of care is Oldrug, a triptan with known moderate side effects. Another treatment, Newdrug, is approved for non-migraine use in cancer patients, but observational studies have shown that it may reduce migraines dramatically. However, Newdrug has potentially severe side effects and so is discouraged for use in treating migraines. The physician will prescribe one drug or the other. The physician enters her patient’s information into the EHR, and an embedded AI system makes a recommendation for treatment. Table 9.1, adapted from our prior work, shows the possible options that could result.6

Table Icon

Table 9.1

Mapping potential physician liability for clinical use of AI.

The table presents eight scenarios of an AI system recommending one of two treatments for patients with chronic migraines – choosing between Oldrug (standard of care) or Newdrug (nonstandard care) – and the possible legal outcomes depending on whether the physician follows or does not follow an AI system’s recommendation.

Malpractice law is at base conservative: because malpractice law typically will find no liability for following the standard of care, the physician will typically not be liable for prescribing Oldrug (the standard of care), whatever else happens. If the drug works well, there is obviously no liability because there is no injury (scenarios 1 and 8). If the drug does not work well (scenarios 3 and 6), the physician will still typically not be liable for injury, because Oldrug is the standard of care.

If, on the other hand, the physician prescribes Newdrug, liability becomes more likely. If Newdrug is the right choice for that patient (the patient was healed), no injury results, and there is no liability (scenarios 4 and 5). If, on the other hand, Newdrug is the wrong choice for that patient, and there is resulting injury, the physician is likely to be liable for actions falling below the standard of care, no matter what the AI said (scenarios 2 and 7).

As the law currently stands, malpractice liability concerns incentivise physicians to follow the standard of care they would have followed before, no matter what the AI suggests; that is, they face incentives to use AI systems essentially as confirmatory advice only. To be clear, this practice may still result in benefits: if AI systems can suggest the standard of care more quickly and easily (or can back that standard up with helpful practice guidelines or other supporting materials), they may still streamline the task of physicians. But to the extent that AI can improve care by suggesting treatments that are better than the standard of care, this approach leaves unrealised some of the value of AI. It also decreases the incentives for adoption; if a significant fraction of the value of medical AI results from nonstandard recommendations, but physicians are unlikely to follow those recommendations, the benefit of AI is decreased, and hurdles to adoption are correspondingly more salient.

However, we note two scenarios in the table (noted with asterisks) that have the most possibility for change or development as AI systems become more prevalent, more accepted, and (ideally) better.

First, in scenario 6, the AI system correctly recommends Newdrug, a treatment at variance with the normal standard of care, and the physician rejects this recommendation, prescribing Oldrug and resulting in patient injury. Under existing law, the physician would most likely be shielded from malpractice liability because she followed the standard of care. But this result is not necessarily static; as AI systems become more prevalent, following AI advice may itself be incorporated into the standard of care, such that ignoring the advice would render a physician liable for resulting injury. While such a shift is possible, malpractice law’s conservatism renders it unlikely in the near future: the commonly applied ‘two schools of thought’ or ‘respectable minority’ doctrines typically shield from liability physicians who follow the practices of a respectable minority, even if those practices are behind the times.

Second, in scenario 7, the AI system incorrectly recommends Newdrug, still outside the normal standard of care, and the physician accepts this recommendation, prescribing Newdrug and resulting in patient injury. Under existing law, the physician would most likely be liable for malpractice liability because she deviated from the standard of care and caused the patient injury. Here, too, the situation could change if following AI recommendations become part of the standard of care. The ‘respectable minority’ or ‘two schools of thought’ doctrines would then work to shield the physician from liability because she would have adhered to the standard of care in following the AI recommendations. As AI becomes more accepted, this shift seems substantially more likely to occur in the short term, though as noted above, caselaw is still substantially undeveloped. Note that the path to this future has pitfalls because until AI systems are accepted as part of the standard of care, physicians deviating from the standard of care are more vulnerable to liability than those simply following older practice patterns.

Whether following AI has become part of the standard of care is likely to be practice area- and application-specific, rather than a general determination across the field of medicine. Such determinations are likely to be influenced by various markers of approbation, such as FDA clearance or approval, recommendations by learned societies and practice guidelines. But the ultimate determination will likely remain a conclusion of what competent physicians actually do, reached by courts after arguments by expert witnesses.

Interesting empirical work suggests that AI may already be shifting into the standard of care – at least in lay conceptions. Kevin Tobia, Aileen Nielson and Alexander Stremitzer undertook a vignette study of 2,000 individuals to simulate the views of lay jurors in the range of scenarios described in Table 9.1 (though focused only on the scenarios where patient injury resulted).7 They found that these potential jurors typically thought physicians should not be liable for following AI recommendations of nonstandard care (that is, AI incorrectly recommending Newdrug), even when injury resulted.8 That is, in scenario 7, though the theoretical application of existing law suggests liability, lay intuition is already that physicians acted acceptably. (Study participants were more ambivalent about whether physicians should face liability for rejecting an AI’s nonstandard recommendation.)9 Lay participants seemed to demonstrate a ‘follow-both’ model, where physicians could act reasonably either by following the standard of care or by following an AI system’s nonstandard recommendation – in either case, study participants found physician actions fairly reasonable.10

The law does not, of course, directly follow lay perceptions of reasonableness on the ground. Lay individuals do serve as jurors, but many factors intermediate between lay perceptions and legal outcomes: experts will testify as to the standard of care; local custom and practice typically matter a great deal; and most cases never reach the jury, whether because judges determine the answer as a matter of law or because cases settle.11 Physician behaviour will matter more than lay perception, but the fact that lay perception is already changing may hint that physician changes to the standard of care could start sooner than one might think.

Some argue that adopting AI into the standard of care will have negative impacts, even as it would increase the use of AI. Michael Froomkin, Joelle Pineau and the late Ian Kerr suggest that if physicians become too deferential to AI because following AI recommendations becomes the standard of care (and rejecting those recommendations, even if in accordance with older practice, invites liability), physicians may lose their skills and knowledge over time.12 If AI performance degrades over time due to expected phenomena such as dataset shift as patient populations and patterns of care change,13 the health system will eventually be in a worse place than it would be without AI – and physicians will have lost the ability to fix things.14 We think this outcome is relatively unlikely, at least in the foreseeable future, partly because the respectable minority doctrine protects older practice patterns, allowing physicians to continue doing what they are doing and avoid using AI.

Finally, once the use of AI becomes part of the standard of care, which will almost certainly happen over some period of time, what will be the standard of care for physicians with respect to the use of AI itself?15 It would be unusual to conclude that a physician should blindly defer to an AI system’s recommendation, whatever it might be; consider a clearly erroneous recommendation to prescribe a very high dose of thalidomide to a pregnant person for the treatment of mild nausea. But how much should physicians defer to AI systems, and how should they interrogate recommendations, especially given the black-box nature of many AI systems and the inability to interrogate the bases beneath decisions? Once AI becomes just another tool in the physician’s toolkit, how must that tool be used? How will FDA approval, or the lack thereof, play into this picture? Some AI systems will escape regulatory review altogether,16 and the question of physician ability to review AI is itself intertwined with the FDA’s ability to regulate AI at all.17 Will the standard of care require such questioning? Even if physicians cannot determine the actual reasoning behind a decision, must they evaluate procedural indicia of quality and reliability, such as how the system was developed or validated?18 Who will supervise those indicia of quality? These issues, too, will unfold in courts and other arenas as injuries happen and lawsuits follow.

3. Institutional Liability

We turn now to institutional liability at the hospital or practice group level – when are they liable for instances when a particular AI use causes an adverse event for the patient? Here it is useful to distinguish two separate theories: derivative liability for the actions of physicians or others; and direct liability for the institution itself.

3.1. Derivative Liability for Hospital Use of AI

Derivative liability depends on first establishing medical malpractice or some other form of liability on the part of the physician or other healthcare provider and then using one of the recognised legal theories that traces that liability to the institution. Under the doctrine of respondeat superior, an ‘employer is subject to liability for torts committed by employees while acting within the scope of their employment’.19 As the Minnesota Supreme Court recently put it, a ‘hospital is vicariously liable for the negligence of its employees where the hospital has control over the actions of the employees’, but ‘[i]f there is a break in the chain of control between employer and employee, the hospital cannot be vicariously liable under the doctrine of respondeat superior’.20 Under this theory, should a patient have a bona fide malpractice claim relating to a hospital employee’s tortious use of AI to direct the patient’s care, and the activities were within the employee’s scope of employment, then liability may flow to the hospital system. A similar theory allows institutional liability if healthcare providers are not formally employed but are subject to sufficient control by the hospital to be treated as employees for liability purposes.21

The composition of most hospital workforces, however, complicates matters. While most nurses are hospital employees and thus the respondeat superior theory may be available as to their negligence, many (but not all) hospital physicians are independent contractors. For independent contractors, respondeat superior will not be available as a theory to reach the hospital at all. But there is a sister theory, often referred to as ‘apparent authority’, which may be available in these cases and applies ‘when a third party reasonably believes the actor has authority to act on behalf of the principal and that belief is traceable to the principal’s manifestations’.22 As the Minnesota Supreme Court recently stated the principle in a case regarding hospitals and independent contractor physicians, the doctrine has two requirements: the principal ‘must have either held the agent out as having authority or knowingly permitted the agent to act on its behalf’; and there was ‘reliance, meaning that the plaintiff was aware of these representations of authority by the principal’.23 As to the reliance element, there is a further question of whether actual reliance is required: whether ‘a plaintiff must show that certain actions would not have been taken but for the appearance of an agent’s authority’, meaning in the hospital context that a plaintiff must show she would not have accepted care as a patient had she known that the physician was not an employee of the hospital system and only an independent contractor.24 The trend has been against requiring this kind of but-for show as to reliance, but not all courts have decided how to apply the test to hospital systems. As applied to our context, should a patient have a bona fide malpractice claim relating to an independent contractor physician’s use of AI to direct the patient’s care, and should the patient be able to show that the hospital presented the physician as its agent and the plaintiff reasonably relied on that representation, then liability may flow to the hospital system.

None of the analysis, thus far, has turned on something distinct about AI; instead, we have just applied the rules governing derivative liability for hospitals to a case where the underlying tort claim relates to AI. When would AI pose distinct liability issues? Scott Schweikart has suggested very briefly that if

a court deems an AI to be fully autonomous (or, if not autonomous, maybe held to be under the dominion of its designers rather than the hospital who purchased and uses it), then holding a hospital vicariously liable for any injury caused by AI will be impossible, as such an autonomous AI will functionally be outside of the principal’s control.25

It is true that the respondeat superior theory breaks down if this eventuality should ever occur, but it is not clear that apparent agency – which does not depend on control – would dissipate.26 The harder question is whether a hospital’s derivative liability could ever be premised on an underlying theory of the AI as an autonomous tortfeasor as opposed to derivative liability traced to the physician who tortiously uses the AI. When the underlying tort is products liability, discussed in greater depth below, it seems more plausible to think of the hospital’s liability as being a species of direct liability than derivative liability such as respondeat superior. These possibilities remain speculative for now.

3.2. Direct Liability for Hospital Use of AI

Apart from the duties of their agents, hospitals also have duties to their patients that can generate direct liability for the hospital as an institution. Such theories are applicable to decisions the hospital makes as to AI, though thus far we have not seen any reported decisions on such fact patterns. Two main hospital direct liability theories might be applied to medical AI in the future: (1) negligent selection/retention and (2) negligent supervision.

The first imposes upon a hospital system a duty to review ‘physicians’ competency and performance history before admission to the medical staff and periodically (typically every two years) thereafter’.27 As the Supreme Court of Wisconsin put it, to recover, a plaintiff must ‘show that the defendant did not exercise reasonable care (that degree of care ordinarily exercised by the average hospital) to determine whether [the physician] was competent’.28 A plaintiff might argue that a hospital system is, in a sense, hiring and not buying an AI system, and that this imposes duties to determine prior errors leading to adverse events from the use of this AI system; to review whether it has received certification, by whom, and the quality of such certification; and perhaps even to determine how it will ‘fit in’ with the existing hospital workforce, much as one would when hiring a live person.29 Moreover, this review cannot be one-and-done, and instead ought to be continuous or at least periodic. Courts may find this theory a step too far in terms of anthropomorphising AI. Even if the theory is endorsed, the test for negligence depends in part on a comparison as to what degree of care is used in these determinations by other hospital systems, which creates a problem at this nascent stage of AI integration in healthcare. But we need not wait for hospital custom to evolve on its own. As one of these authors has put it: ‘policymakers could try to move hospitals’ standard of care for implementing black-box algorithms toward one that would involve procedural tools to make sure that algorithms are well validated and competently developed before implementation.’30

Negligent supervision is, by contrast, more controversial even as to flesh-and-blood hospital personnel. Rather than imposing a duty at the time of hiring and periodic review, this theory ‘assumes contemporaneous supervision of daily treatment decisions as they are made’.31 While several decisions have alluded to such a duty, it has largely been in dictum.32 The duty has also been imposed in cases ‘of gross negligence in which the departure from medical standards is so blatant that it is possible to attribute to hospital administrators’ constructive knowledge of the error in progress’.33 Especially as to more opaque forms of medical AI, we think courts will be more sceptical of negligent supervision theories applying. As a predictive matter, it seems unlikely that courts will impose a duty upon hospitals to supervise each AI recommendation and reliance thereon by a physician ‘as they are made’ in addition to the negligent selection/retention duties and whatever derivative liability exists.

Beyond these two primary theories, one more is worth mentioning, though it is more penumbral: while ‘hospitals are typically not liable for defects in the products they provide and/or sell, they may have a duty to nonnegligently evaluate the quality of those products and may be liable for failures of products that they fail to evaluate’.34 One of the few cases in this line, Parker v St. Vincent Hospital, involves a suit against a hospital relating to implanted bilateral artificial temporomandibular joint replacement devices that were made by Vitek, Inc. in a surgery performed at the hospital.35 The New Mexico Court of Appeals held that it was not ‘appropriate to impose strict products liability on hospitals with respect to a defectively designed medical product selected by the treating physician’.36 In the same opinion, it did suggest that an action for negligence against the hospital could be valid, although it hemmed and hawed and ultimately did not decide the scope of that duty. Nevertheless, some of its reflections are germane to the question we address:

Should a hospital conduct its own research study regarding the efficacy and safety of implants; should it review the medical literature for pertinent findings by researchers elsewhere; should it monitor the experience of patients who receive implants at the hospital? [. . .]

On the record before us, however, we cannot confidently make that determination. We are unable to determine whether imposition on a hospital of any particular duty to investigate the safety of implants or other medical devices promotes or retards public policy. If a duty to investigate would require considerable effort and expense by hospitals, resulting in higher costs for medical care, but would add little to patient safety, it would be unwise to impose the duty. Safety would not be enhanced, for example, if the hospital were merely duplicating efforts by the FDA, particularly given that the hospital would have a far smaller data base to work from, which could lead it to draw inaccurate inferences. On the other hand, if, as alleged by an expert witness provided by Plaintiffs, hospitals already have a duty under federal law to conduct the sort of investigation Plaintiffs would require, then there may be little reason not to impose liability on a hospital that injures a patient because of failure to perform that duty with due care. On remand these matters can be explored and a record prepared that is adequate for the court to make a proper judgment on the existence and scope of any duty to investigate.37

We think these same problems appear in spades with any attempt to specify the scope of a theory of negligent evaluation of medical AI. Hospitals will, especially in the case of more opaque medical AI, lack the expertise to conduct their own evaluations. Moreover, unlike with some other medical devices, many medical AI systems will not have gone through any premarket review by the FDA, which could be treated as a seal of approval. Our instinct, though it is only an instinct, is that it may be better to channel these cases through the gates of either the negligent selection/retention tort or products liability, rather than recognising an additional tort theory in the medical AI world analogous to Parker.

4. Developer Liability

In addition to potential physician and institutional liability, there is also a pressing question of how and whether developers of faulty medical AI can be held liable under current tort law.38 In the following, we first explain the difference between negligence and strict liability. We then discuss FDA regulation and its potentially preemptive effects.

4.1. Difference Between Negligence and Strict Liability

Suppose a physician uses a medical AI in the treatment of a Black patient with cancer. The AI recommends an incorrect nonstandard drug dosage that the physician follows, and the patient’s condition worsens. As seen above, the physician may likely be held liable for causing injury to the patient.39 However, it turns out that the reason for the faulty AI recommendation was that the model was mainly trained on data from White patients. While the physician in this hypothetical scenario will likely incur liability for a bad patient outcome under current law,40 one key question still remains to be answered: Could the developer of the medical AI be likely held liable for negligence because the model was predominantly trained on data from White patients?

To establish a prima facie case for negligence, the plaintiff (here, the Black patient) must prove – by a preponderance of the evidence (that is, more than 50 per cent) – four elements: duty, breach, causation and damage.41 A successful negligence claim thus requires that the defendant (here, the AI developer) owes a legal duty to the patient and that this duty was accidentally breached, which caused the patient’s injury.

Up to the 1910s, injured consumers of flawed products were often unable to successfully sue manufacturers for negligence because they could not establish a duty of care due to a lack of contractual privity.42 Nowadays, courts no longer require privity for the existence of such a duty, and assume it.43 However, to recover for negligence, consumers still need to establish a breach of this duty, injury to them and actual and proximate causation between the breach and the injury.

The challenges faced by injured consumers in proving fault and thus succeeding in a negligence claim against manufacturers eventually led to the introduction of strict products liability in the 1960s.44 Section 402A of the Restatement (Second) of Torts (1965) states:

  • (1) One who sells any product in a defective condition unreasonably dangerous to the user or consumer or to his property is subject to liability for physical harm thereby caused to the ultimate user or consumer, or to his property, if
    • (a) the seller is engaged in the business of selling such a product, and
    • (b) it is expected to and does reach the user or consumer without substantial change in the condition in which it is sold.
  • (2) The rule stated in Subsection (1) applies although
    • (a) the seller has exercised all possible care in the preparation and sale of his product, and
    • (b) the user or consumer has not bought the product from or entered into any contractual relation with the seller (emphasis added).

Product liability – which is generally considered a strict liability of manufacturers for product defects – has evolved over time, and courts have established three types of product defects, namely (1) design defects, (2) manufacturing defects and (3) marketing defects.45 While a design defect is inherent and already exists before manufacturing the product, a manufacturing defect is a physical departure from the intended product’s design and occurs during its production or construction.46 Marketing defects refer to inadequate instructions or failures to warn consumers about possible risks associated with the use of the product.47

For instance, in our hypothetical example, a claim for a marketing defect may be given if the labelling of the AI did not include a warning that the model may likely not give reliable or correct recommendations when used in non-White patients. Obviously, such a model that has not been trained on a diverse patient population should not be placed on the market in the first place and may thus also trigger a design defect suit.48 When considering healthcare software, however, most courts have so far been hesitant to hold developers liable under product liability theories.49 The reason for this seems to be the assumption that such software is a clinical decision support tool that only gives recommendations and that it is the physician who ultimately decides.50 In other words, software has been interpreted as a service rather than a product.51 Thus, under current case law, it is likely that injured patients will have a hard time successfully suing developers of medical AIs under product liability. But a court’s shift to product liability would not be inconceivable in the future, considering that high-performing deep learning networks are increasingly being deployed in medicine, which are impossible or difficult for humans to understand (so-called black boxes).52

An important distinction here is between a medical AI system that received marketing authorisation from the FDA and a medical AI system that is marketed without the need for FDA review. This distinction may be relevant for future court decisions regarding whether product liability applies in cases of healthcare software. The FDA does not regulate the practice of medicine (that is, services), but it does regulate medical devices, and if healthcare software is classified as such in a particular case, product liability is not outside the realm of possibility in the future.53

The distinction also matters because regulatory actions by the FDA may preempt state law, insulating some AI manufacturers from state-law tort claims. We now turn to FDA preemption.

4.2. FDA Preemption and Its Interaction with FDA Regulation

FDA preemption is a controversial legal theory that shields manufacturers of certain products from tort claims. Express preemption means that the federal statute – here, the US Federal Food, Drug, and Cosmetic Act (FDCA) – explicitly includes a preemption provision that all or some state law is displaced.54 There is also implied field preemption in cases where the statutory language is not express, but the displacement of state law can be implied by Congress’s intent to occupy the area exclusively.55 Lastly, there is conflict preemption, such as in cases where state and federal requirements contradict each other, and a party cannot comply with both.56

In the context of medical AI, FDCA Section 521 provides for preemption. It states:

  • (a) General Rule.—Except as provided in subsection (b), no State or political subdivision of a State may establish or continue in effect with respect to a device intended for human use any requirement—
    • (1) which is different from, or in addition to, any requirement applicable under this Act to the device, and
    • (2) which relates to the safety or effectiveness of the device or to any other matter included in a requirement applicable to the device under this Act.
  • (b) Exempt Requirements.—Upon application of a State or a political subdivision thereof, the Secretary may, by regulation promulgated after notice and opportunity for an oral hearing, exempt from subsection (a), under such conditions as may be prescribed in such regulation, a requirement of such State or political subdivision applicable to a device intended for human use if—
    • (1) the requirement is more stringent than a requirement under this Act which would be applicable to the device if an exemption were not in effect under this subsection; or
    • (2) the requirement—
      • (A) is required by compelling local conditions, and
      • (B) compliance with the requirement would not cause the device to be in violation of any applicable requirement under this Act.57

Consequently, FDCA Section 521 generally displaces state law with respect to medical devices for human use. The term ‘medical device’ is defined in FDCA Section 201(h)(1). Some AI-based products are classified as medical devices (AI-based medical devices) under FDCA Section 201(h)(1) because they are ‘intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease’. Others fall outside the scope of the medical device definition and thus a priori of FDCA Section 521 and its express preemption. In particular, software functions pursuant to FDCA Section 520(o) are not considered medical devices under FDCA Section 201(h)(1). For example, many medical AIs that support specific clinical decisions by providing transparent recommendations to healthcare professionals meet this exception and, thus, are not classified as medical devices.58

Suppose an AI-based product is classified as a medical device. In that case, it is regulated by the FDA and usually needs to undergo premarket review.59 Depending on their risk level (that is, low, moderate, high), AI-based medical devices are categorised into three classes (that is, Class I, Class II and Class III).60 The device class is usually pivotal in determining the applicable premarket pathway.61 There are three common premarket submissions for medical devices:

  1. 510(k) Premarket Notification.
  2. De Novo Classification Request.
  3. Premarket Approval (PMA).62

In general, the 510(k) (clearance) pathway is applicable for Class I or Class II medical devices that are not exempt from premarket submission and that are substantially equivalent to a so-called predicate – a legally marketed device.63 The De Novo Classification Request applies to new low- to moderate-risk medical devices that have no predicate device and provides a pathway to classify such devices into Class I or Class II.64 Finally, PMA is for most Class III (highest risk) medical devices and is the most strict premarket submission type, requiring valid scientific evidence that the device is reasonably safe and effective for the intended use.65 There are also some Class III medical devices that may only require a 510(k).66

Almost all AI-based medical devices on the US market received 510(k) clearances.67 For example, Philips’s Precision Position was FDA-cleared in June 2021.68 This device uses AI algorithms to precisely position a patient before a CT scan.69 Only a few marketed AI-based medical devices received authorisation through the De Novo process, and only three AI-based medical devices have so far received PMA approval.70 For example, Oxehealth Vital Signs is an AI-based medical device that analyses video signals and estimates a patient’s heart, pulse, respiratory, and breathing rates.71 This device, which is incorporated into a vision-based patient management and monitoring platform called Oxevision, received marketing authorisation from the FDA via the De Novo pathway in March 2021.72

The fact that most marketed AI-based medical devices are 510(k)-cleared is crucial in limiting the scope of preemption: the US Supreme Court ruled in Medtronic, Inc. v Lohr (1996) that FDCA Section 521(a) did not preempt the state law tort claims of the plaintiffs, Lora Lohr and her husband, in the failure of Lora Lohr’s Medtronic pacemaker.73 In this case, the Medtronic pacemaker was a Class III medical device that underwent a 510(k) and was found substantially equivalent.74 In particular, the Supreme Court argued that ‘[s]ince the § 510(k) process is focused on equivalence, not safety, substantial equivalence determinations provide little protection to the public’, and thus device manufacturers would need to defend themselves against state-law negligent design claims.75 It also refused to accept preemption for Lohrs’ manufacturing and labelling (failure to warn) claims because these requirements were too general and not concerned with respect to a particular medical device.76

In Riegel v Medtronic, Inc. (2008), the US Supreme Court ruled that FDCA Section 521(a) preempts ‘common-law claims challenging the safety or effectiveness of a medical device marketed in a form that received premarket approval from the FDA’.77 This time, the court considered a Medtronic catheter, a Class III medical device that received a PMA approval.78 The catheter ruptured in the coronary artery of Charles Riegel during heart surgery, and Charles Riegel and his spouse, Donna Riegel, filed a suit against Medtronic, alleging that the device’s label, design and manufacture violated New York common law.79 In contrast to Lohr, the Medtronic catheter underwent a PMA (rather than 510(k)) in this case, and thus Medtronic provided valid scientific evidence that the device was reasonably safe and effective for the intended use.

Consequently, manufacturers of an AI-based medical device that receives PMA approval are likely protected against state-law tort claims which challenge the device’s safety or effectiveness.80 However, only three AI-based medical devices have so far gone through PMA.81 Since most marketed AI-based medical devices were cleared via the 510(k) pathway, the vast majority of manufacturers are not shielded from such state-law tort claims under the preemption clause in FDCA Section 521(a). In Lohr and Riegel, the US Supreme Court did not address the issue of whether FDCA Section 521(a) preempts state-law tort claims that challenge the safety or effectiveness of marketed medical devices that underwent the De Novo process. On the one hand, since the PMA is the most rigorous type of premarket submission and requires premarket approval, judges may argue that manufacturers of De Novo devices are not protected from state-law liability. On the other hand, if the AI-based medical device is classified into Class II via the De Novo pathway, it requires special controls and manufacturers must ‘provide reasonable assurance of safety and effectiveness and a description of how the special controls provide such assurance’.82 Thus, one might convincingly argue that manufacturers of marketed Class II devices that underwent the De Novo process may be protected from state-law tort claims that challenge the device’s safety or effectiveness through FDCA Section 521(a) in accordance with Lohr and Riegel.83 Until the Supreme Court weighs in, we will not know for sure. In general, with the rapid development of medical AI, the regulatory framework may change in the future, including the preemption landscape.84 Stakeholders should thus watch this space.

5. Conclusion and Moving Forward

We close with a few thoughts about liability for medical AI writ large. First, and broadest, this is a space in flux; we have laid out the workings of generally applicable law, but there remains substantial uncertainty as to how these factors will fall into place once cases start coming to courts – and legislatures and regulators could always step in to change things substantially. The most obvious changes are to the standard of care, where the use of AI is likely to become an accepted part of the standard care over time but likely at different rates in different parts of medical practice. But the way in which the FDA (or other regulators) regulates FDA, and potential implications on liability, could also easily change.

Although outside our scope here, the latest developments in the European Union are also close to watch since they can contain valuable lessons for the United States. The European Commission proposed two EU Directives in September 2022: a Directive on liability for defective products85 and an AI Liability Directive.86 While the Directive on liability for defective products is primarily aimed at ‘provid[ing] an EU-level system for compensating people who suffer physical injury or damage to property due to defective products’,87 the AI Liability Directive’s goal is to address especially the issue that ‘current national liability rules, in particular based on fault, are not suited to handling liability claims for damage caused by AI-enabled products and services’.88 The two new proposed Directives go hand in hand with the AI Act89 to ensure, among other things that AI systems placed on the Union market and used are safe and respect existing law on fundamental rights and Union values.90 The AI Act is expected to come into force in late spring or early summer 2024.

Second, the discussions of liability above refer only to the initial allocation of liability; individual actors can change that allocation by contract, including through indemnification agreements or through the purchase of insurance.91 Two implications flow from this. Most directly, more informed parties can signal quality by assuming liability for problems – for example, Digital Diagnostics carries medical malpractice liability insurance for its IDx-DR diabetic retinopathy diagnosis system and assumes liability for injuries arising from the system.92 Less directly, insurers may serve as a separate, quasi-independent verifier of AI system quality, where a positive evaluation could become necessary for insurance coverage.

Third and finally, the dynamic nature of AI and injury further complicates the picture of liability. How liability is allocated and assigned when injury occurs shapes the behaviour of developers, institutions and individuals93 – but so does the amount of injury in the first place.94 If the integration of AI systems into healthcare practice lowers the overall level of injury substantially, the picture changes. Consider a stylised pre-AI system where physicians, hospitals and product developers equally share liability for injuries that occur. If 150 injuries occur, each is liable for the equivalent of 50 injuries. Imagine the addition of AI makes hospitals liable for all the actions of physicians (who blame systems for errors they would previously be responsible for) but also cuts the rate of injuries by 60 per cent. Now, out of 60 injuries, the hospital is liable for the equivalent of 40 (developers 20, physicians 0). Even though liability allocation has changed, the hospital is still better off because the overall rate of injuries has decreased. This decrease is also, of course, a socially desirable outcome. One could, however, change the story to get a socially bad outcome; if liability allocation followed the same pattern but AI only decreased the rate of injury by 40 per cent, the hospital would be liable for the equivalent of 60 out of 90 total injuries – and, if it knew this result ex ante, could be expected to resist the implementation of the AI system, in a blow to overall welfare. All of this is to say that liability for individual actors tells only part of the story; the efficacy of the AI systems can profoundly shift the overall picture.

As we noted at the outset, liability for medical AI presents a complicated landscape, with many players, doctrines and interactions. Understanding the moving pieces is essential both for individual actors in the system and for policymakers considering how best to shape the adoption of high-quality AI moving forward.

Notes

1

This work was funded by a Novo Nordisk Foundation Grant for a scientifically independent International Collaborative Bioscience Innovation & Law Programme (Inter-CeBIL programme – grant no. NNF23SA0087056). Sara Gerke’s work was also partially funded by the European Union (Grant Agreement no. 101057321). However, the views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

2

Caitlin Owens, ‘Doctor Acquisitions Spiked Amid the Pandemic’ (Axios, 29 June 2021) <https://www​.axios.com​/doctor-hospitals-acquisitions-coronavirus-pandemic-7bfdaf84-72fd-4870-b4fc-619a208edcf9.html> accessed 19 February 2022.

3

Frank Griffin, ‘Artificial Intelligence and Liability in Health Care’ (2021) 31 Health Matrix 65, 95.

4

W Nicholson Price, Sara Gerke and I Glenn Cohen, ‘Potential Liability for Physicians Using Artificial Intelligence’ (2019) 322 JAMA 1765.

5

ibid.

6

ibid.

7

Kevin Tobia, Aileen Nielsen and Alexander Stremitzer, ‘When Does Physician Use of AI Increase Liability?’ (2020) 63(11) Journal of Nuclear Medicine 1.

8

ibid 9.

9

ibid.

10

ibid.

11

W Nicholson Price, Sara Gerke and I Glenn Cohen, ‘How Much Can Potential Jurors Tell Us About Liability for Medical Artificial Intelligence?’ (2021) 62 Journal of Nuclear Medicine 15.

12

A Michael Froomkin, Ian Kerr and Joelle Pineau, ‘When AIs Outperform Doctors: Confronting the Challenges of a Tort-Induced Over-Reliance on Machine Learning’ (2019) 61 Arizona Law Review 33.

13

Samuel G Finlayson and others, ‘The Clinician and Dataset Shift in Artificial Intelligence’ (2021) 385 New England Journal of Medicine 283–86.

14

Froomkin, Kerr and Pineau (n 12).

15

W Nicholson Price II, ‘Medical Malpractice and Black-Box Medicine’ in I Glenn Cohen and others (eds), Big Data, Health Law, and Bioethics (CUP 2018).

16

W Nicholson Price II, Rachel Sachs and Rebecca S Eisenberg, ‘New Innovation Models in Medical AI’ (2022) 99 Washington University Law Review 1121.

17

21st Century Cures Act 2016, s 3060(a) (United States).

18

Price II (n 15).

19

Restatement (Third) Of Agency § 2.04 (2006) (United States).

20

Popovich v Allina Health System 946 NW 2d 885, 891 (MN 2020) (United States).

21

Scott v SSM Healthcare St Louis 70 SW 3d 560 (MO Court of Appeals 2002) (United States).

22

Restatement (Third) Of Agency § 2.03 (2006) (United States).

23

Popovich v Allina Health System (n 20) 895 (cleaned up).

24

ibid 895–96.

25

Scott J Schweikart, ‘Who Will Be Liable for Medical Malpractice in the Future? How the Use of Artificial Intelligence in Medicine Will Shape Medical Tort Law’ (2021) 22(2) Minnesota Journal of Law, Science & Technology 1, 16.

26

See Cefaratti v Aranow 141 A 3d 752, 609 (CT 2016) (noting that apparent agency can apply even without control) (United States).

27

Mark A Hall and others, Health Care Law and Ethics (9th edn, Wolters Kluwer 2018) 445; while most courts recognise the theory of liability, at least one has rejected it: see Paulino v QHG of Springdale, Inc 386 SW 3d 462 (AR 2012) (United States).

28

Johnson v Misericordia Community Hosp 301 NW 2d 156 (WI 1981) 739 (United States).

29

Price II (n 15) 303 (‘hospitals could be liable for negligently choosing, implementing, and using black-box medical systems’).

30

ibid.

31

Hall and others (n 27) 445.

32

ibid 446 (citing Thompson v Nason Hosp 591 A 2d 703 (PA 1991) (United States)).

33

ibid 445–46. Some courts have rejected the tort theory outright: ibid 445 (citing Essig v Advocate BroMenn Medical Center 33 NE 3d 288 (IL Appellate Court 2015) (United States)).

34

Price II (n 15) 303.

35

Parker v St Vincent Hosp 919 P 2d 1104, 1106 (NM Court of Appeals 1996) (United States).

36

ibid 1107.

37

ibid 1112.

38

Griffin (n 3) 78.

39

See section 1 above; Price, Gerke and Cohen (n 4).

40

See ibid.

41

For more information on the negligence concept see eg, John L Diamond, Lawrence C Levine and Anita Bernstein, Understanding Torts (6th edn, Carolina Academic Press 2018) ch 3.

42

See eg, MacPherson v Buick Motor Company 111 NE 1050 (NY Court of Appeals 1916) (United States).

43

James Underwood, Tort Law: Principles in Practice (3rd edn, Wolters Kluwer) 709.

44

See Greenman v Yuba Power Products 59 Cal 2d 57 (CA 1963) (United States); see for the initial development eg Escola v Coca Cola Bottling Co. of Fresno 150 P 2d 436 (CA 1944) (United States).

45

Legal Information Institute (LII), ‘Products Liability’ (Cornell Law School) <https://www​.law.cornell​.edu/wex/products_liability> accessed 1 May 2024; Sara Gerke, Timo Minssen and I Glenn Cohen, ‘Ethical and Legal Challenges of Artificial Intelligence-Driven Healthcare’ in Adam Bohr and Kaveh Memarzadeh (eds), Artificial Intelligence in Healthcare (Academic Press 2020) 314; for case law see eg, American Tobacco Co. v Grinnell 951 SW 2d 420 (TX 1997) (United States).

46

LII (n 45); Restatement (Third) of Torts: Products Liability, § 1, cmt a (1998) (United States).

47

LII (n 45).

48

See also Barbara J Evans and Frank Pasquale, ‘Product Liability Suits for FDA-Regulated AI/ML Software’ in I Glenn Cohen and others (eds), The Future of Medical Device Regulation: Innovation and Protection (CUP 2022).

49

W Nicholson Price II, ‘Artificial Intelligence in Health Care: Applications and Legal Implications’ (2017) 14 The SciTech Lawyer 10, 11.

50

ibid 12.

51

Evans and Pasquale (n 48).

52

For more information on black boxes see eg, W Nicholson Price II, ‘Black-Box Medicine’ (2015) 28 Harvard Journal of Law & Technology 419; W Nicholson Price II, ‘Regulating Black-Box Medicine’ (2017) 116 Michigan Law Review 421; Boris Babic and others, ‘Beware Explanations From AI in Health Care’ (2021) 373 Science 284; Boris Babic and Sara Gerke, ‘Explaining Medical AI Is Easier Said Than Done’ (STAT, 21 July 2021) <https://www​.statnews​.com/2021/07/21/explainable-medical-ai-easier-said-than-done> accessed 1 May 2024.

53

Evans and Pasquale (n 48).

54

Peter Barton Hutt, Richard A Merrill and Lewis A Grossman, Food and Drug Law: Cases and Materials (4th edn, Foundation Press 2014) 292.

55

ibid.

56

ibid.

57

Federal Food, Drug, and Cosmetic Act (FDCA) s 521 (emphasis added) (United States).

58

ibid s 520(o)(1)(E); for more information on this exception see also FDA, ‘Clinical Decision Support Software – Guidance for Industry and Food and Drug Administration Staff’ (2022) <https://www​.fda.gov/media​/109618/download> accessed 1 May 2024; Price II, Sachs and Eisenberg (n 16).

59

For example, the FDA exercises enforcement discretion over some low-risk medical devices, see eg FDA, ‘Policy for Device Software Functions and Mobile Medical Applications – Guidance for Industry and Food and Drug Administration Staff’ (2022) <https://www​.fda.gov/media/80958/download> accessed 1 May 2024.

60

See FDA, ‘How to Study and Market Your Device’ (US Food & Drug Administration, 12 October 2023) <https://www​.fda.gov/medical-devices​/device-advice-comprehensive-regulatory-assistance​/how-study-and-market-your-device> accessed 1 May 2024.

61

ibid.

62

ibid.

63

For more information on the 510(k) pathway, see eg FDA, ‘Premarket Notification 510(k)’ (US Food & Drug Administration, 5 December 2023) <https://www​.fda.gov/medical-devices​/premarket-submissions​/premarket-notification-510k> accessed 1 May 2024; see also FDCA (n 57) s 513(i).

64

For more information on the De Novo process, see eg FDA, ‘De Novo Classification Request’ (US Food & Drug Administration, 4 October 2022) <https://www​.fda.gov/medical-devices​/premarket-submissions​/de-novo-classification-request> accessed 1 May 2024; see also FDCA (n 57) s 513(f)(1) and (2).

65

FDA, ‘How to Study and Market Your Device’ (n 60); for more information on PMA, see eg FDA, ‘Premarket Approval (PMA)’ (US Food & Drug Administration, 16 May 2019) <https://www​.fda.gov/medical-devices​/premarket-submissions​/premarket-approval-pma> accessed 1 May 2024; see also FDCA (n 57) s 513(a)(1)(C).

66

FDA, ‘Premarket Notification 510(k)’ (n 63).

67

FDA, ‘Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices’ (US Food & Drug Administration, 6 December 2023) <https://www​.fda.gov/medical-devices​/software-medical-device-samd​/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices> accessed 1 May 2024.

68

Letter from the FDA to Philips Healthcare (Suzhou) Co., Ltd., ‘K203514’ (17 June 2021) <https://www​.accessdata​.fda.gov/cdrh_docs/pdf20/K203514.pdf> accessed 1 May 2024.

69

ibid.

70

ibid.

71

Letter from the FDA to Oxehealth Limited, ‘DEN200019’ (26 March 2021) <https://www​.accessdata​.fda.gov/cdrh_docs/pdf20/DEN200019.pdf> accessed 1 May 2024.

72

ibid. Oxehealth, ‘FDA Grants Oxehealth Vital Signs De Novo Clearance’ (PR Newswire, 31 March 2021) <https://www​.prnewswire​.com/news-releases​/fda-grants-oxehealth-vital-signs-de-novo-clearance-301259496.html> accessed 1 May 2024.

73

Medtronic, Inc. v Lohr (1996) 518 US 470 (United States).

74

ibid.

75

ibid.

76

ibid.

77

Riegel v Medtronic, Inc. 552 US 312 Syllabus (2008) (United States).

78

ibid.

79

ibid.

80

For an in-depth analysis of the challenges of the PMA process, preemption and AI-based medical devices, see Charlotte Tschider, ‘Medical Device Artificial Intelligence: The New Tort Frontier’ (2021) 46 BYU Law Review 1551.

81

See FDA, ‘Premarket Approval (PMA)’ (n 65).

82

FDCA (n 57) s 513(f)(2)(A)(v) (United States).

83

See also James M Beck, ‘FDA De Novo Device Classification Process & Preemption’ (Drug & Device Law, 10 December 2018) <https://www​.druganddevicelawblog​.com/2018​/12/fda-de-novo-device-classification-process-preemption​.html> accessed 1 May 2024.

84

See eg Sara Gerke, ‘Health AI for Good Rather than Evil? The Need for a New Regulatory Framework for AI-Based Medical Devices’ (2022) 20 Yale Journal of Health Policy, Law & Ethics 433.

85

Commission, ‘Proposal for a Directive of the European Parliament and of the Council on Liability for Defective Products’ COM (2022) 495 final.

86

Commission, ‘Proposal for a Directive of the European Parliament and of the Council on Adapting Non-Contractual Civil Liability Rules to Artificial Intelligence (AI Liability Directive)’ COM (2022) 496 final.

87

COM (2022) 495 (n 85) 1.

88

COM (2022) 496 (n 86) 1.

89

European Parliament Legislative Resolution of 13 March 2024 on the Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts (COM(2021)0206 – C9-0146/2021–2021/0106(COD) (AI Act).

90

ibid Recital 1.

91

Ariel D Stern and others, ‘AI Insurance: How Liability Insurance Can Drive the Responsible Adoption of Medical Artificial Intelligence’ (2022) 3 NEJM Catalyst Innovations in Care Delivery.

92

Michael D Abràmoff, Danny Tobey and Danton S Char, ‘Lessons Learned About Autonomous AI: Finding a Safe, Efficacious, and Ethical Path Through the Development Process’ (2020) 214 American Journal of Ophthalmology 134, 139.

93

George Maliha and others, ‘Artificial Intelligence and Liability in Medicine: Balancing Safety and Innovation’ (2021) 99 The Milbank Quarterly 629.

94

Tobia, Nielsen and Stremitzer (n 7).

Copyright Edward Elgar Publishing.

This work is licensed under the Creative Commons Attribution-NonCommercial-No Derivatives 4.0 License

Bookshelf ID: NBK613216PMID: 40245228DOI: 10.4337/9781802205657.ch09
Contents

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this page (259K)

In this Page

Similar articles in PubMed

See reviews...See all...

Recent Activity

ClearTurn OffTurn On

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...