This chapter considers the current state of play of the governance ecosystem for AI in the UK. It aims to provide an overview of where key regulators are in relation to the challenge and opportunity AI presents, as well the likely direction of travel in the years ahead. Starting from the UK government’s National Strategy for AI, alongside that of the NHS, the chapter then explores how individual regulatory bodies for health have themselves responded, the role of technical standards and guidance and the planned changes to the regime for regulating medical devices in the UK. Key themes that emerge are the extent of current flux, the persistence of formal regulatory fragmentation contrasted with informal coordination, reliance on soft law to drive regulatory change, and optimism in the face of the need to resolve the plural set of goals identified as relevant here.

1. Introduction

The governance of artificial intelligence (AI) and machine learning (ML) in the United Kingdom’s (UK) healthcare systems naturally engages a range of stakeholders, bodies and regulatory frameworks. Many aspects of what follows are unavoidably influenced by broader civic dialogue over the general governance of AI. This chapter offers a critical snapshot of the state of relevant governance frameworks in the UK, flagging regional, transnational or international developments as needed.

The following section sets out the high-level background relevant to AI governance in the UK. The subsequent thematic sections then consider various specific issues in greater detail, to the extent currently possible. The first addresses the architecture of governance for AI in health, exploring the institutional division of responsibilities and responses. The second addresses the role of ethical and technical guidance. The third considers the medical device regime as it applies to AI. A conclusion then draws together the strands and considers the values that appear to inhabit the governance ecosystem. Much of what follows draws on government consultations, policy documents and position papers issued by the relevant regulatory bodies and other interested stakeholders, rather than being able to rely on clear and settled legal frameworks. Thus, while this affords good insight into the likely direction of travel in the UK, much remains to emerge by way of such settled legal structures.

2. AI Governance In UK Healthcare: The Policy Background

This section identifies the broad priorities, strengths, weaknesses and suggested responses in the two most pertinent, wide-scale assessments of AI in the UK: the UK government’s National AI Strategy and the UK National Health Service’s (NHS) own strategy. Both deal with the issues of regulation and governance, the latter directly in the health context. They set the stage for much of what will follow.

From the broadest vantage point, in 2021 the UK government set out a National AI Strategy.¹ This established a ten-year vision for the cross-sectoral deployment of AI. Composed of three pillars, the Strategy both considers the need to foster the grounding infrastructure AI requires (Pillar 1) and ensures broad deployment across regions and sectors (Pillar 2). While it does not speak directly to healthcare, the Strategy identifies general themes for UK AI governance (Pillar 3).² The increasingly widespread deployment of AI is one of the key assumptions on which the National Strategy was based,³ yet it was agnostic on whether revision of the governance ecosystem is required. Indeed, it certainly envisages a world where existing structures are left undisturbed and then supplemented to respond appropriately to the specific challenges AI creates in a given sector.⁴ Within that existing governance ecosystem, the National Strategy highlights the role of soft law and industry standardisation alongside more formal institutional, command-and-control forms of regulation.⁵ In particular, it emphasises the importance of coordinated technical standards at the outset of AI’s deployment in a given field as a means of embedding key values, complemented in due course by assurance practices that certify, validate and audit AI throughout its lifecycle.⁶ Thus, the National Strategy sets the broad parameters for the evolution of AI governance, suggesting that it will be sector-specific and reliant on softer regulatory intervention rather than hard law.

While the National Strategy is precisely that, the government was also alive to the importance of existing and emerging relevant global and regional frameworks. Notably, given the UK’s recent withdrawal from the European Union (EU), this included the EU’s proposed AI Act,⁷ albeit the Strategy also notes the UK’s correlate opportunity to create an innovative governance regime of its own.⁸ The hope for some collaboration on the trans- or international stages is intended to ‘prevent divergence and friction between partners, and guard against abuse of this critical technology’.⁹ Whether friction can be avoided or minimised will now, as set out below, likely depend on the specific institutional networks that exist (or which are now fostered) across particular facets of the governance ecosystem, rather than relying on substantive uniformity as a default. Given the recurrent rhetoric around supporting innovation underpinned by a world-leading governance regime, the hope is that the UK’s contributions to shaping international norms will succeed and prove decisive – a goal that will be simple to assess in due course.¹⁰

A series of white papers, the most recent from March 2023, at the time of writing developed the National Strategy, confirming many of its themes. Several of these elaborations are noteworthy. First, particular governance challenges of relying on manifold sector-specific responses to AI have been identified: lack of clear and transparent ownership of governance responsibility in a given sector; inconsistency of action and powers between regulatory agencies; and the general risk of an existing governance ecosystem that does not fully and appropriately respond to the risks and challenges of AI.¹¹ Second, five key principles with cross-sectoral relevance have been identified. These are: (1) safety, security and robustness; (2) appropriate transparency and explainability; (3) fairness; (4) accountability and governance; and (5) contestability and redress.¹² These will, at least initially, be put on a non-statutory basis, with implementation the responsibility of existing regulators in each sector.¹³ Likewise, to the extent that a settled definition of AI is possible, two core functional capabilities of such systems have been highlighted as underpinning the need for specific regulatory responses: the adaptivity of AI systems and the autonomy with which they may make consequential decisions.¹⁴ These factors will determine whether a system is regarded as AI-based for the relevant regulatory consideration.¹⁵ Finally, the same concern for flexibility and future-proofing the UK’s AI governance is evident in the adoption of a proportionate approach that focuses on regulating the use of a given system, rather than assigning risk levels to entire sectors.¹⁶ While the government acknowledges the benefits and risks of AI in healthcare, given the decision to rely on sector-specific governance, the relevance of this national vision lies in the values highlighted as central to the AI context. Naturally, the importance and effect of these high-level values relies on their reception and implementation by key regulatory bodies.

Turning to the NHS, we find broadly similar concerns. While operational governance within the NHS takes place on a devolved basis, such that there is an individual NHS architecture for each of England, Scotland, Wales and Northern Ireland, the largest of these bodies, NHS England, has been the most active to date in the broader development and deployment of AI in healthcare.¹⁷ The most relevant work here has been driven by a body set up in 2019 to foster the safe use of new technologies within the NHS: NHSX (from early 2022 part of the NHS Transformation Directorate). Preceding the UK’s National Strategy, an NHSX-led report in 2019 comprehensively considered the role and governance of AI in health.¹⁸ While there was a clear recognition of the ‘significant ethical and safety concerns’ around the use of AI in health,¹⁹ the report concluded optimistically, matching the tone of the later national visions in terms of the transformative potential of AI and the UK’s role as a ‘global leader in ethical AI’.²⁰ The report provides critical insights into the direction of the relevant governance ecosystem in healthcare. Empirical engagement with developers revealed that a ‘current complex governance framework for AI technologies is perhaps limiting innovation and potentially risking patient safety’.²¹ As discussed below, this is hardly surprising. Partly due to this lack of clarity, almost half of developers were not seeking ethical approval before developing relevant technologies; nor were a similar proportion seeking or planning to seek certification as a medical device, despite the likely need to do so.²² The need to develop the governance ecosystem for AI in health emerged clearly as one of the key ‘pain points’ for future interventions to address, ranking alongside generally upskilling those working in the field.²³ The importance and role of both ethical guidance and legal regulation were highlighted,²⁴ ultimately suggesting that while the relevant legal frameworks might often be ‘perceived as a barrier to the implementation and adoption of AI in healthcare’, this stems from a lack of coordination between regulators and relevant statutory bodies rather than the regulation itself, again suggesting that evolution rather than revolution may characterise the UK’s response.²⁵ The report identified five challenges that need to be addressed in response:²⁶

First, in the current landscape no one body/unit is responsible for the overall process making it difficult to ensure coordination between regulators. Second, regulation can often be wrongly interpreted on the ground, particularly regarding regulation around data. Third, in some very specific instances, the regulation itself is not fit-for-purpose. The letter of the law would require people to go through such cumbersome processes that regulators follow the ‘spirit of the law’ instead. Fourth, in some cases the remit of regulators is unclear or overlapping, which means that no one is responsible for policing a specific regulatory requirement. No regulator has direct oversight over the quality of the data used to train algorithms, meaning that no one is responsible for preventing bias in algorithmic tools. Finally, there are uncertainties about how to regulate certain aspects of AI.

Although the common root for developing AI interventions lies in data access for model training, NHSX also reported significant confusion regarding the legal basis for accessing medical data and the applicability of the health research regulatory regime to relevant software.²⁷

Taken together, there is a common vision for the years ahead. Both the National and the NHS Strategies suggest that the UK can act as a global leader, which the Bletchley Declaration would appear to evidence.²⁸ Both recognise the importance of the governance ecosystem and the role that both legal and ethical frameworks play. Both support a sector-specific approach to regulating AI in health. Yet, an attempt to fully capture the governance ecosystem for AI in healthcare is, presently, an exceedingly difficult exercise, given the fragmentation of UK healthcare regulation.²⁹ Despite sharing common goals, existing governance architectures focused on institutional, professional and stakeholder interests may struggle to achieve coherence and clarity. Indeed, the quality of governance is obviously undermined if developers are uncertain about whether they are caught within it at all. In analysing this fragmented regulatory landscape, emphasis will be placed on where specific consideration has already been given to AI, even if the consequences of such consideration are still in motion.

3. The Evolving Governance Architecture

The extent to which a sector-specific approach impacts the governance ecosystem for AI becomes clear when one maps the various regulatory bodies involved. Indeed, key stakeholders themselves struggle to comprehend the relevant structures and relationships between them throughout the lifecycle of AI technology in health. To do so, NHSX required not only a literature review, but a series of interviews and workshops with relevant stakeholders.³⁰ This uncovered five obvious regulatory bodies, four statutory bodies and ‘a multitude of other bodies with a role in this field’.³¹ This alone presents a central challenge for any national policy goals and rightly generates much sympathy for developers seeking to navigate this.

The central regulatory bodies identified were: the Care Quality Commission (CQC); the Information Commissioner’s Office (ICO); the General Medical Council (GMC); the NHS Health Research Authority (NHS HRA); and the Medicines and Healthcare products Regulatory Agency (MHRA). Collectively, their areas of interest span the development, deployment and monitoring of relevant AI technologies but no single one holds comprehensive responsibility for overseeing this lifecycle. Equally, the locus of interest differs across these bodies. In the case of the ICO, controlling or processing personal data (of which health data is just one example) engages their remit.³² By contrast, the CQC is only seized of those engaging in a list of specified health services.³³ Likewise, the GMC’s remit is limited to the relevant registered healthcare professionals for whom (re-)training required to integrate AI into practice and broader ethical implications are relevant.³⁴ This section considers some of the specific responses observed across these key bodies.

Flagged by NHSX as part of the necessary foundation for developing the UK’s regulatory framework,³⁵ the CQC has adopted a regulatory sandbox concerning AI. To date, this sandbox has completed two rounds, focused respectively on digital triage algorithms and ML in diagnostics.³⁶ Regulatory sandboxing has been deployed in cognate fields,³⁷ and has the potential to contribute effectively to the development of regulatory norms and structures in the healthcare sector.³⁸ The CQC regards these sandboxes as ‘a way of working proactively and collaboratively to understand new types of health and social care service, agree what good quality looks like, and develop our approach to regulation’.³⁹ The sandboxes aimed to concretise what constitutes ‘good’ in these contexts, offering recommendations in each report, having engaged service users, providers and developers alongside other key regulators and bodies. As the CQC acknowledged,⁴⁰ although the formal, architectural fragmentation identified above maintains natural barriers between agencies,⁴¹ the substance of discussion central to re-assessing respective approaches may need to be more centralised and holistic, even if administrative competencies remain disparate. Regulators may need to engage a broader range of stakeholders and account for connections between one regulatory framework and another. Three broader insights may be drawn from the CQC’s sandbox. The first, at least in the context of both reported rounds, was the substantive lacunae identified in the existing regime. In some cases, the development and deployment of AI systems has blurred the previously clear line between health devices and health services such that the CQC considers regulation of at least some technology suppliers inevitable.⁴² Nevertheless, this only highlights that the CQC lacks the necessary technical expertise.⁴³ In others, developments revealed a general absence of necessary uniform assurance structures.⁴⁴ The second was the practical insight into a sector where much is unknown, despite the sensitivity of the interests involved. One admirable advantage of the sandbox approach is its ability to reveal how the regulatory structures are perceived. Regarding diagnostic services, the CQC clarified that suppliers needed to better clarify for healthcare providers that relevant systems actually relied on ML, and what this meant they were capable of.⁴⁵ Likewise, the CQC was not capturing the necessary information to allow proper assessment of services.⁴⁶ Assessing whether health services are meeting the required quality is understandably harder when those services themselves are unaware of the regulated elements their service involves. Reflecting NHSX’s insight, there were ‘varying levels of compliance with software and device regulations’.⁴⁷ The third was the extent to which regulatory innovation or response is likely constrained by the need for formal action by other bodies. Both sandbox reports highlight the entangled nature of the UK’s health governance ecosystem, particularly concerning standards, accreditation and evidence.⁴⁸ It becomes impractical, if not impossible, to instantiate goal-oriented, amorphous terms such as ‘quality’ or ‘safety’ in the absence of technical validation of the regulated systems. Likewise, spurring innovation is at risk when the regulation’s signalling function is too vague to act upon. The practical consequences of the sandbox are still emerging, but it appears to have been successful in identifying general guidance in the respective contexts and in highlighting broader recurring themes across the sector.

While the CQC has openly adopted a sandbox approach, some of the same features can be viewed elsewhere, particularly the reliance on collaborative development.⁴⁹ The first example was the free-to-use HealthTech Connect database set up in April 2019 by the National Institute for Health and Care Excellence (NICE), the body charged with producing quality standards and assessing the effectiveness of health interventions.⁵⁰ Flagged as one means of overcoming regulatory uncertainties among developers,⁵¹ this project aimed to support the earlier identification of necessary requirements and avoid duplication by connecting developers with relevant decision-making bodies. The project has now transitioned to the NHS Innovation Service, where the Accelerated Access Collaborative offers similar support by connecting developers with regulatory and commissioning services within the NHS.⁵²

Likewise, the NHS HRA – primarily responsible for ensuring awareness of and compliance with the regulatory and ethical governance in developing data-driven technologies – is working towards a range of services designed to engage and support developers. The NHS HRA’s AI and Digital Regulations Service serves as a single access point for developers and prospective users seeking guidance on regulatory pathways. As with the Accelerated Access Collaborative, this involves a partnership between key regulatory bodies such as NICE, the CQC and the MHRA.⁵³ Similarly, the HRA aims to streamline the technological review process such that developers can engage earlier and more effectively with research approvals. As noted, NHSX’s analysis revealed the extent to which it was ‘currently quite hit and miss whether or not developers seek ethical approval at the beginning of the development process with an almost 50/50 split between those that did and those that did not’.⁵⁴ In a similar vein, it identified confusion around the data access stage and whether particular developer activities – such as training software using medical data – ought to be regarded as research necessitating HRA approval.⁵⁵ This might reflect that those developing potentially impactful AI systems may lack a background or previous experience in the health sector. If so, the already complex regulatory requirements for those within the sector are likely to prove more challenging for newcomers. As such, efforts by key regulators to streamline and simplify their processes are necessary and have a high potential to change developer practice. Again, the HRA’s work has relied in part on qualitative research into developer practice and that of patients, clinicians, academics and the HRA’s staff and volunteers.⁵⁶ This work is already yielding results, with a new combined review process for clinical trials of investigational medicinal products (CTIMPs) now available as a pilot, while a broader coordinated assessment process for medical devices is being explored. The goal is to have the MHRA, the research ethics committee and the HRA work in parallel based on a single application, with CTIMPs offering a shorter overall period for review.⁵⁷ What ultimately stands out here is that the HRA’s efforts are largely structural rather than substantive, clarifying the borders of the respective regulatory pathways.

There is a clear impetus towards coordination, given the need to foster clarity around the regulatory journey. Nevertheless, relying on a ‘you-tell-us-we-tell-you’ approach highlights that such clarity remains aspirational and that – as the HRA candidly noted – those who staff regulatory agencies ‘need a deeper understanding of AI if we’re to speed up the approvals process’.⁵⁸ The regulatory development model may, thus, be fairly characterised as largely collaborative rather than top-down, command-and-control.

4. The Role Of Technical Standards And Guidance

One clear aspect of the UK’s regulatory response in the health sector is the role of soft law and pre-emptive guidance as a key part of the governance ecosystem. Such instruments support either the ethical development and implementation of AI systems, or effective engagement with regulatory requirements. The possibility to draw on, feed into or otherwise engage with international or transnational equivalents is particularly clear here, with various international statements, principles and frameworks available.⁵⁹

The role of ethical frameworks was explicitly highlighted alongside the legal regulatory frameworks by NHSX,⁶⁰ emphasising assessment of whether interventions should be undertaken at all and assigning formal gatekeeping to legal regulation. Ethical guidance was valued as one means of creating a system that proactively addresses potential AI risks.⁶¹ Central to UK developments is the Code of Conduct for Data-Driven Health and Care Technology, first issued in 2018 to reflect and amplify the data-related work of the Nuffield Council on Bioethics,⁶² and a range of existing regulatory and NHS guidance.⁶³ Upon issue, three of the principles were novel and required further implementing work: principle 7 (on algorithmic explainability), principle 8 (on generating evidence of effectiveness) and principle 10 (on commercial strategy). Regarding the Code’s intended uses, NHSX’s strategy was to provide an online portal for developers to respond to a workbook version of the Code and engage in a process of self-assurance.⁶⁴ Given the black-box algorithmic function of AI systems,⁶⁵ principle 7 was intended to help developers understand what was expected of them and – in the hope of fostering transparency and trust – to provide clear guidance on its practical implementation.⁶⁶ The Code was updated in 2021 to a set of guidance issued by the Department of Health & Social Care on good practice for digital and data-driven health technologies, sharing many of the same core principles.⁶⁷ Feedback on the Code suggested that its practical advice was valued, and the new guidance has expanded on this. While the ethical imperative in this guidance is clear, and the concerns specific to AI are highlighted where relevant, many of the key underlying goals emphasised – ‘transparency, accountability, safety, efficacy, explicability, fairness, equity, bias’⁶⁸ – apply equally to the non-AI contexts the guidance covers. These ethical imperatives must be viewed in light of the guidance’s openly instrumental aim to ensure that by fostering the expected good practice by design, the NHS will be able to realise the benefits such technologies offer. Similar guidance can also be issued by particular regulators, with one example being the ten principles jointly identified by the US FDA and the UK’s MHRA under the banner ‘Good Machine Learning Practice for Medical Device Development: Guiding Principles’ (GMLP).⁶⁹ While, in comparison, these are less obviously intended to support navigation of regulatory pathways, such efforts allow identification of underlying ethical and practical norms and demonstrate the potential for international feedback loops in this area.⁷⁰

Soft guidance is crucial in healthcare, including in the AI context, where such documents provide necessary ballast to largely generalised legal frameworks and an optimistic rhetoric of innovation and impact. Again, though, the work is fragmented across institutions. As the relevant national body, the British Standards Institute (BSI) interacts extensively with international equivalents, through the ISO, and the relevant European bodies, with the aim to register as the relevant AI-notified body under the EU’s AI Act. The National Strategy highlighted the BSI’s capacity to drive and contribute to the global development of technical standards as part of the broader reassessment of national governance architecture.⁷¹ Additionally, the role of standards issued by the Data Coordination Board in ensuring clinical safety,⁷² along with various ISO standards, was highlighted.

One area in which novel standards have emerged is NICE’s work on clinical effectiveness and economic utility. Validation from NICE is a higher bar than being able to place a device on the market. While NICE focuses on effectiveness rather than safety,⁷³ it represents a gold standard of validation within the UK’s healthcare framework. Robust evidence is needed, given the potential consequences: a mismatch between a product’s value claims and the supporting evidence can be fatal.⁷⁴ Following a period of beta access in 2018, in March 2019 NICE issued an Evidence Standards Framework (ESF) for Digital Health Technologies (DHTs) to underpin principle 8 of the 2018 Code of Conduct for Data-Driven Health and Care Technology.⁷⁵ For the purposes of this chapter, this 2019 version of the ESF was only relevant to Al health technologies that relied on fixed algorithms, but an updated version issued in August 2022 now defines a data-driven digital health technology as one that ‘contains algorithms that were trained using patient data or datasets. These algorithms could be adaptive, meaning they change over time, or fixed’ and that ‘uses decision thresholds or cut-off values (such as for diagnosing a condition or triaging patients for different treatments) that were created using patient data or datasets’.⁷⁶ DHTs are divided into three tiers (A, B and C) depending on the intended use and potential risk, with the requirements scaling proportionately depending on the tier.⁷⁷ Tier C, into which most medical devices and in vitro diagnostics will fall, is subdivided into four classification groups based on the intended use, in line with the International Medical Device Regulators Forum (IMDRF) classification and the MHRA’s Software as a Medical Device programme (SaMD), discussed below.⁷⁸ Systems underpinned by AI/ML might be found in any of the three tiers. The ESF sets out 21 standards, grouped unevenly across five categories: Design factors; Describing value; Demonstrating performance; Delivering value; and Deployment considerations.⁷⁹ Not all standards must be met by DHTs in all tiers, but many apply to all. Furthermore, in instantiating these principles there is, predictably, heavy reliance on the standards and technical documentation from other bodies in the governance ecosystem, with developers often directed to draw from such to demonstrate compliance with the relevant ESF standard. A specific addition to the updated 2022 ESF is identifying a minimum viable subset of the ESF, which represents ‘early deployment standards’ that can assess whether DHTs are suitable for evidence-generation programmes designed to support pilot systems early in their development and deployment. This subset – encompassing 16 of the standards – and such programmes aim to support technologies that might otherwise struggle to generate evidence required to meet the full ESF at that early stage.⁸⁰ While there has been some scepticism at the high evidential bar set by the full ESF – and accordingly emphasis on the role of the early deployment framework⁸¹ – a public consultation in April 2022 on the initial version of the ESF found that 55 per cent of respondents felt they would not create a barrier to innovation, and most respondents agreed that the early deployment subset would meet developer needs.⁸²

Reflecting the UK’s regulatory fluidity, the original exclusion of adaptive DHTs ensured revision would be needed.⁸³ Even the updated ESF presumes that the regulatory framework for DHTs, ‘particularly those with adaptive algorithms’ will need further updating. On that front, close attention is being given to the MHRA’s work,⁸⁴ with the 2022 update aligning the digital healthcare classifications with the MHRA’s.⁸⁵

To accommodate AI systems, the ESF largely relies on goal-oriented standards. Take, for instance, the universal Standard 4 on considering health inequalities and bias mitigation. Here, the ESF counsels that a company engaging with the framework should ‘describe any actions taken in the design of the DHT to mitigate against algorithmic bias that could lead to unequal impacts between different groups of service users or people’.⁸⁶ Given the lack of clarity on the shared language for auditing such risks,⁸⁷ developers might understandably desire more detailed guidance. Something more specific is provided regarding Standard 16 on measuring performance over time. Here, additional goal-oriented requirements are provided for adaptive DHTs (or where system updates can be expected), highlighting the importance of agreeing on future plans for any updates or retraining, the basis for such retraining and the processes to measure performance over time.⁸⁸ Yet, as this suggests, to the extent that the ESF does provide certainty, it may be characterised – as in the NHS’s strategy – as providing ‘a common reference standard for discussions between innovators, investors and commissioners’ rather than a commonality of the outcome that such discussions will yield across different products.⁸⁹ While NHSX suggests that the ESF contributes to standardising evaluation of AI in health,⁹⁰ it is, again, standardisation of form, not necessarily content. The goal of greater specificity has been flagged as key to advancing NICE’s work.⁹¹ One might conclude that – at least for AI – the framework strikes a collaborative rather than imperative note and permits an evaluative ambiguity, rather than something more prescriptive. Such flexibility may ultimately be advantageous in a fast-moving, diverse field of technologies. Finally, beyond the immediate audience of developers, NICE explicitly envisages the ESF being used for purchasing decisions by health institutions and having a broader sectoral impact as the market develops.⁹²

Efforts to revise standards in light of AI demonstrate the UK’s ongoing process of regulatory reassessment, and highlight the potential for international collaboration.⁹³ Existing work produced a series of white papers from the BSI and the Association for the Advancement of Medical Instrumentation, flagging the need to develop an international and common language for standards terminology concerning AI in health.⁹⁴ Here, the challenge lies in the manifold uses and forms of AI in health, along with the tension between developing novel standards and adapting existing frameworks. One means of creating an ‘overarching “umbrella” standard’ is through mapping national approaches onto the IMDRF’s Essential Principles. The aim would be to identify ‘the key principles that need to be addressed across the AI life cycle, from the perspectives of developers and the healthcare system’.⁹⁵ Such international collaboration would still rely on national bodies to settle the specific requirements, with consequent scope for divergence in practice. Yet, given the UK’s current preference for soft law to affect necessary changes where possible, such alignment of principles may represent the greatest degree of uniformity possible.

5. A Medical Device Regime For Software And AI

Of the key regulatory bodies, a central role falls to the MHRA’s regime for medical devices.⁹⁶ Given that the definitional focus of a medical device lies in its intended purpose, it is clear that many devices incorporating AI are likely to be so regarded.⁹⁷ Considering the confusion among developers about the relevance of the MHRA’s regime to their product,⁹⁸ how AI is accommodated within it becomes a pressing legal and practical matter.

In assessing the UK position, the current state of flux in the regime due to the UK’s withdrawal from the European Union is unavoidable.⁹⁹ While the existing rules under the Medical Devices Regulations 2002 (MDR 2002) remain in force,¹⁰⁰ they will be replaced in due course with a new regime intended to apply from mid-2025.¹⁰¹ Although a new UK-specific product marking has been provided (the UK Conformity Assessed marking, UKCA),¹⁰² the existing CE marking also remains valid. However, the UKCA is not presently recognised in the EU (or, given the technicalities of withdrawal, the Northern Ireland market) nor are the relevant UK notified bodies designated by the MHRA (now referred to as Approved Bodies) recognised by the EU for granting a CE mark.¹⁰³ The extent to which a novel regulatory regime for medical devices as a whole will emerge in the future will clearly impact AI-based technologies as well. Nevertheless, it is possible to identify general and specific characteristics at this point that will likely prove relevant.

Medical devices in the UK have not required pre-marketing assessment by the MHRA, but must now be registered with them before being placed on the market.¹⁰⁴ The onus falls on manufacturers to ensure compliance with the essential requirements, through approval from relevant notified bodies.¹⁰⁵ Assessment of these essential requirements focuses largely on a device’s technical performance.¹⁰⁶ In navigating this process, the classification of a medical device is critical, with the UK legislation still mapping to the four-fold schema of the original European directives.¹⁰⁷ Given the withdrawal, the Medicines and Medical Devices Act 2021 (MMDA) now supplements the MDR 2002, laying the groundwork for future formal regulatory innovation. The MMDA specifies three objectives that must be considered when introducing new regulations: the safety of devices, the availability of devices and the likelihood that the UK is seen as a favourable place for researching, developing, manufacturing or supplying medical devices.¹⁰⁸ If novel regulations will impact patient safety, they can only be introduced ‘if the Secretary of State considers that the benefits of doing so outweigh the risks’.¹⁰⁹ Thus, these keystone objectives provide high-level direction while the MMDA concurrently provides the potential to de-align the UK regime from the EU. While working to ensure the favourability of the UK’s regulatory regime dovetails neatly with the rhetoric of a pro-innovation mindset to AI regulation,¹¹⁰ concerns have naturally been raised over this being placed on a par with patient safety or how the anticipated balancing will be undertaken.¹¹¹ Indeed, using far-reaching delegated powers to amend the existing regulatory framework risks complicating an already challenging situation.¹¹²

Alongside the MMDA, the government’s response to a late-2021 public consultation on the future of medical device regulation suggests the direction of travel, with three matters worth noting. The first are the five policy pillars on which the UK’s regulatory framework for medical devices will develop. These are: strengthening the MHRA’s role in patient safety; making the UK a focus for medical device innovation; addressing health inequalities throughout a device lifecycle; proportionate regulation that builds on synergies with EU and global standards; and working to make the UKCA a world-leading mark.¹¹³ Second is a practical insight into whether and where the UK will align with other regimes. On topics such as the definition of medical devices and the substance of essential requirements, the EU’s MDR serves as the relevant lodestone, with close alignment planned.¹¹⁴ On others, such as the classification of medical devices, aligning with ‘best international practice’ appears to be the goal.¹¹⁵ Across the consultation, no single other regime emerges as influential over any other, albeit respondents frequently flagged the importance of alignment with the EU’s MDR. The third matter is the explicit consideration of AI within the consultation. Beyond the MHRA’s existing work, explored below, the desire to create a regulatory framework ‘fit for the future for the regulation of software and artificial intelligence as medical devices’ was highlighted as the third keystone objective in the MMDA.¹¹⁶ Likewise, AI was used as a justification for developing alternative routes to market, to enable ‘game changing innovation’.¹¹⁷ In sum, while the general framework for medical devices is currently transforming, the importance of AI is clear within that, and legal pathways now exist to support regulatory innovation in response to it.

Beyond this general landscape, in late 2021 the MHRA embarked on its Software as a Medical Device programme of consultation (SaMD).¹¹⁸ The matter of AI as a medical device (AIaMD) is considered a subset of this broader programme, with many overlapping challenges and solutions.¹¹⁹ The programme aims to review existing regulations and support confidence in the overall safety and assurance behind such devices.¹²⁰ Given the ‘urgent need’ to ensure the regime is fit for purpose, this is a welcome step.¹²¹ The programme was initially split across 11 work packages (reduced to 8) and was also considered as part of the broader MHRA consultation in 2022. Those individual work packages address specific points in the regulatory cycle – such as definitional and classificatory issues, and pre- and post-market requirements – as well as broader AI-specific thematic concerns such as interpretability and adaptability (work packages 9, 10 and 11).¹²² Some packages have begun generating outputs and, alongside the government’s responses to that 2022 consultation, greater clarity can be expected to emerge as these work packages progress. As might be hoped for, much of the work undertaken has necessarily involved collaboration with other regulatory and guidance-producing bodies.¹²³ Likewise, the programme acknowledges that departing from ‘international consensus’ may create additional market burdens and, thus, positions this process of national regulatory reassessment as within and as a means to ‘drive forward international consensus in this area’.¹²⁴

As to the present state, the following points are noteworthy. The definition of software is consistent with that of the EU,¹²⁵ with guidance issued to help developers navigate regulatory conformity.¹²⁶ Indeed, mirroring the CQC’s sandbox approach to safely support innovative technologies, the MHRA is exploring the potential for an ‘airlock’ classification where the risk profile of a SaMD is unclear.¹²⁷ Likewise, the concept of ‘placing on the market’ will likely have a context-specific meaning for SaMD, given the potential availability of AI-based health apps through websites.¹²⁸ The gravity of international alignment is again evident in assessing whether additional essential requirements are necessary for SaMD.¹²⁹ A majority of respondents to the consultation were keen to see the UK’s regime align with GSPR 17 of the EU framework. Additional essential requirements will be introduced, focused on themes such as cyber security and data protection, but as a sub-division rather than a separate set of essential requirements for SaMD. Further clarification of the essential requirements for AIaMD, if needed, will be provided through guidance on satisfying the relevant essential requirement rather than creating something novel. Regarding post-marketing requirements, although the MHRA’s Yellow Card scheme generally applies,¹³⁰ predetermined change control plans (PCCPs) were strongly endorsed to clarify confusion over notification requirements where software is updated.¹³¹ By contrast, this will require a ‘clear legislative foothold’ and, while initially voluntary, such plans may become mandated.¹³² The MHRA will now advance this work through its aptly named ‘Project Ship of Theseus’. Finally, while the MHRA assumes the SaMD programme is generally relevant for AI, the 2022 consultation sought to clarify whether any further AI-specific statutory changes would be beneficial. Only a few respondents favoured statutory change over the continued flexibility of guidance, with most using the opportunity to flag general areas of concern regarding AIaMD.¹³³ For fear of being overly prescriptive, there will be no definition for AIaMD or further statutory requirements beyond SaMD, and any required work will be carried out through changes in the relevant guidance. One such example was developing clinical performance evaluation methods for SaMD (encompassing AIaMD) along the lines of the IMDRF’s procedures.¹³⁴ Beyond changes in the regulatory framework for devices itself, the Regulatory Horizons Council’s independent advisory report on AIaMD in 2022 specifically highlighted the importance of long-term funding for SaMD/AIaMD regulation and the need to ensure internal capacity and capability among the staff of regulators.¹³⁵ Overall – and confirming a theme across the UK’s responses – while some legislative change can be anticipated,¹³⁶ much of what is likely to emerge from SaMD will take the form of guidance on clarifying the approach to these technologies rather than revising the framework in light of them.¹³⁷ Equally, reflecting the preference for sector-specific regulation in the UK’s National Strategy, SaMD/AIaMD will likely be treated as a sub-domain of an otherwise generalised regulatory framework.

6. Conclusion: The State And Values Of Governance

In 2019, NHSX described overcoming the regulatory pain points as a ‘long-term and evolving project’.¹³⁸ The contents and consequences of this project are still in flux and whether the UK will achieve its goals remains to be seen. This is perhaps an inevitable process for any system responding to AI’s use in healthcare and is compounded here by the consequences of recent political developments. Yet, if developers remain uncertain about whether they must navigate those pathways¹³⁹ – with obvious speed-to-market incentives in avoiding them – greater regulatory clarity may matter for little. As such, although it is foolhardy to suggest where this protean state will settle, some key matters are clear.

While regulating AI has been left to specific sectors, beyond that there is less formal regulatory specification than one might assume necessary. This is reflected in the reliance on guidance and standards that marks many of the nascent responses here. In a fast-moving field, regulatory intervention on such a level may well prove easier to produce and amend and can, as practice shows, provide direct means of involving key non-institutional stakeholders, whether developers, users, patients or otherwise. The disadvantages, however, are equally clear. Such guidance, particularly where goal-oriented, may lack either clarity or certainty. This may simply be an unavoidable reality, where the natural limits of interpretability are reached and an iterative, collaborative approach between regulators and developers is required to secure approval. Where ongoing updates or adaption are inevitable, such proactive regulatory collaboration throughout a product lifecycle may be expected.

AI appears unlikely to spur reassessment of the institutional fragmentation that characterises the UK’s governance ecosystem in health; developers cannot yet expect a one-stop shop. While superficially inelegant, this reflects existing institutional inertia in the sector and highlights the myriad stakeholders justly involved.¹⁴⁰ Yet, this formal institutional separation masks an increasing degree of collaboration in response to AI, even if coordination rather than monolithic conformity may be the highest aspiration.

The gravity of the EU and other international frameworks appears unavoidable. Many substantive choices for the UK will be taken in the shadow of decisions taken elsewhere and likely tipping market pressure towards coordination. National governance regimes will encounter developers who may be substantially less bounded, and a country like the UK, that prides itself on offering world-leading healthcare, makes a rod for its own back if it demurs to provide what is impactfully on offer elsewhere. In articulating why a given product fails to pass regulatory muster, regulators who plough their own furrow will need to clearly articulate this difference, given regulatory comparison is inevitable.

Turning from the mechanics of governance, it is worth reflecting on the values at play. It is, of course, correct that AI engages ‘unique challenges’ for governance,¹⁴¹ but some of these are questions about what constitutes good regulation. Here, institutional fragmentation makes it difficult to identify uniform conceptual or normative underpinnings. Naturally, patient safety is a key priority, along with the risk of bias or AI-generated health inequalities.¹⁴² Yet, just as common are themes of supporting innovation, creating a proportionate, agile regulatory environment and the desire to make the UK a global lead in the regulatory sphere. Perhaps this reflects that, currently, many of the relevant materials are policy statements and consultations, which must inevitably hew to the rhetoric of an optimistic state. It may also reflect the natural plurality of goals that combining health interests and AI creates. Given the developing nature of the field, what stands out by its present absence is the clear articulation of how to resolve tensions between these many goals. As is so often the case, the natural hurdles that regulation creates are often perceived as barriers to progress.¹⁴³ As Brownsword compellingly suggests, what may be necessary (most necessary) in the context of AI in health are the means and measures to help regulators ensure that a given technological solution does not transgress the basic conditions for human agency itself.¹⁴⁴ That is not to suggest consideration of these tensions is altogether absent, or that it will not take on greater visibility as the current molten state settles in the future. It is simply to note that the regulatory responses considered can also be understood as an aggregate rhetoric reflecting a set of explicit and tacit values. Viewed as such, the defining theme here is that of an optimist who foresees only inevitable progress and a world in which pressing individual concerns – important as they are – can be appropriately and consistently balanced with broader communitarian interests. Reliance on soft ethical and legal guidance at least provides some means for engaging these concerns, but responsibility is primarily cast onto developers, with the risk of devolving the vital substance and responsibilities of regulation to the regulated themselves.<TX>

Notes

1

Department for Science, Innovation and Technology, National AI Strategy (Cm 525, 2021).

2

ibid 50–62.

3

ibid 7.

4

ibid 53–54.

5

ibid 56–57, what the Strategy itself refers to as ‘traditional’ regulation.

6

ibid 57–58.

7

ibis 55.

8

ibid 54.

9

ibid 55. See also Department for Science, Innovation and Technology, A Pro-Innovation Approach to AI Regulation (Cm 815, 2023) 7.

10

National Strategy (n 1) 55. Indeed on this front it is worth noting that in November 2023 the UK convened a global summit on general AI Safety engaging a substantial number of leading national systems—many of which are covered in this very volume—and resulting in the so-called Bletchley Declaration. This suggests at least some momentum, driven by the UK in this instance, towards coordinated development here, with healthcare noted as a relevant sector in which such concerns naturally inhere. Department for Science, Innovation and Technology, ‘The Bletchley Declaration by Countries Attending the AI Safety Summit, 1–2 November 2023’ (GOV​.UK , 1 November 2023) (The Bletchley Declaration) <https://www​.gov.uk/government​/publications​/ai-safety-summit-2023-the-bletchley-declaration​/the-bletchley-declaration-by-countries-attending-the-ai-safety-summit-1-2-november-2023> accessed 5 May 2024.

11

Department for Science, Innovation and Technology, Establishing a Pro-Innovation Approach to Regulating AI (Cm 728, 2022) 6–7.

12

A Pro-Innovation Approach (n 9) 6.

13

ibid.

14

Establishing a Pro-Innovation Approach (n 11) 9; such an approach was confirmed in the subsequent white paper, see A Pro-Innovation Approach (n 9) 22.

15

ibid.

16

A Pro-Innovation Approach (n 9) 25. Albeit there may be circumstances where a more blunt approach to categorisation of risk will be called for. The Bletchley Declaration (n 10) noted that although many of the relevant risks involved with AI call for international cooperation, it was also plausible that a given national response, ‘could include making, where appropriate, classifications and categorisations of risk based on national circumstances and applicable legal frameworks.’

17

That is not to say that some devolved nations have not considered the matter themselves: see The Scottish Government, Scotland’s Artificial Intelligence Strategy (2021) <www​.scotlandaistrategy.com> accessed 5 May 2024; likewise, the deployment of AI is not limited to England, see the work of the Scottish Radiology Transformation Programme, ‘Artificial Intelligence’ (Scottish Radiology Transformation Programme) <www​.radiology.scot.nhs​.uk/projects/artificial-intelligence> accessed 5 May 2024. Regulatory bodies, by contrast, are usually fully national in scope, covering the whole of the UK.

18

Indra Joshi and Jessica Morley, ‘Artificial Intelligence: How to Get it Right. Putting Policy into Practice for Safe Data-Driven Innovation in Health and Care’ (NHSX 2019).

19

ibid 17.

20

ibid 72.

21

ibid 22.

22

ibid.

23

ibid 23.

24

ibid 27.

25

ibid 37.

26

ibid.

27

ibid 40. Although answers to these questions are asserted in the report itself, there was little obvious basis for definitively doing so.

28

ibid 13, 72; The Bletchley Declaration (n 10).

29

See, generally, David Gomez (ed), The Regulation of Healthcare Professionals (2nd edn, Sweet & Maxwell 2019).

30

Joshi and Morley (n 18) 37.

31

ibid.

32

For the ICO’s current work in the context of AI, see Information Commissioner’s Office, ‘Artificial Intelligence’ (ICO) <https://ico​.org.uk/for-organisations​/uk-gdpr-guidance-and-resources​/artificial-intelligence> accessed 5 May 2024.

33

For the so-called regulated activities, see Health and Social Care Act 2008 (Regulated Activities) Regulations 2014, sch 1.

34

At the time of writing, the GMC is yet to issue specific guidance on AI for doctors, but it did confirm in a 2020 exchange with the Committee on Standards in Public Life that existing principles across its various guidance documents are responsive to the increasing use of AI.

35

Joshi and Morley (n 18) 43.

36

CQC, ‘Getting to the Right Care in the Right Way – Digital Triage in Health Services’ (2020) (Digital Triage); CQC, ‘Using Machine Learning in Diagnostic Services’ (2020) (Diagnostic Services).

37

Sofia Ranchordas, ‘Experimental Regulations for AI: Sandboxes for Morals and Mores’ (2021) Morals + Machines 86; see also the potential for an MHRA sandbox (there described as an ‘airlock’) approach to generating the necessary regulatory evidence for software as a medical device where innovative software meets a critical unmet need (MHRA, ‘Software and AI as a Medical Device Change Programme – Roadmap’ (GOV​.UK , 14 June 2023) <https://www​.gov.uk/government​/publications​/software-and-ai-as-a-medical-device-change-programme​/software-and-ai-as-a-medical-device-change-programme-roadmap> accessed 5 May 2024).

38

Emily Leckenby and others, ‘The Sandbox Approach and its Potential for Use in Health Technology Assessment: A Literature Review’ (2021) 19 Applied Health Economics and Health Policy 857.

39

Digital Triage (n 36) 2.

40

ibid 9 (‘The learning from the early stages of the process had a much broader application than just CQC’s regulation, and it showed that CQC working alone can only have limited impact in driving improvements in care in this area’).

41

An example of this fragmentation may be seen in the Diagnostic Services report (n 36), which lists 11 actors as constituting the existing regulatory framework.

42

Diagnostic Services (n 36) 2.

43

ibid 8, 12; Digital Triage (n 36) 4 (‘While CQC is responsible for regulating providers of healthcare services in a clinical setting, we have struggled to delineate clear boundaries between the suppliers of technology solutions and the health services that use them’).

44

Diagnostic Services (n 36) 2, noting the need for guidance around clinical validation of algorithms and how AI should be implemented in clinical pathways. See also discussion, at 11, of the validation challenge where systems are biased and how that should be monitored post-market.

45

ibid.

46

Digital Triage (n 36) 5.

47

ibid 6. This might be thought to mirror the ‘varying degrees of clinical oversight, governance, and support for innovators and teams that want to use these technologies’ identified among healthcare providers, see Diagnostic Services (n 36) 9.

48

Digital Triage (n 36) 7; Diagnostic Services (n 36) 6–7, 11.

49

Such emphasis on engagement has been a common aspiration on the NHS’s part: see Matthew Gould, ‘Regulating AI in Health and Care’ (NHS Digital, 11 February 2020) <https://digital​.nhs.uk​/blog/transformation-blog​/2020/regulating-ai-in-health-and-care> accessed 5 May 2024; see also BSI and AAMI, Machine Learning AI in Medical Devices (White Paper, 2020) 12–13.

50

Hannah Crouch, ‘NICE Launches Online Portal to Help Support Health Technologies’ (digitalhealth, 30 April 2019) <https://www​.digitalhealth​.net/2019/04/nice-online-portal-health-technologies-support/> accessed 8 May 2024.

51

Joshi and Morley (n 18) 41.

52

‘NHS Accelerated Access Collaborative’ (NHS England) <https://www​.england.nhs.uk/aac> accessed 5 May 2024.

53

At the time of writing, this service is in a live beta trial: see ‘Understanding Regulations of AI and Digital Technology in Health and Social Care’ (NHS Digital Regulations) <https://www​.digitalregulations​.innovation.nhs.uk/> accessed 5 May 2024.

54

Joshi and Morley (n 18) 22.

55

ibid 40. The desire for greater clarity around the definition of health research resurfaced in the HRA’s work: ‘It needs to be easier at an earlier stage to identify which AI and data-driven activities are research, and exactly what approvals will be required’, see ‘Streamlining Data-Driven Research (SDDR)’ (NHS Health Research Authority, 31 October 2022) <https://www​.hra.nhs.uk​/planning-and-improving-research​/research-planning​/how-were-supporting-data-driven-technology​/sddr/> accessed 5 May 2024.

56

As the HRA notes, for its collaborative aims, here ‘the users are the co-designers’, see ‘The Users are the Co-Designers’ (NHS Health Research Authority, 28 July 2022) <https://www​.hra.nhs.uk​/about-us/news-updates​/users-are-co-designers/> accessed 5 May 2024.

57

‘Combined Review’ (NHS Health Research Authority, 27 February 2024) <https://www​.hra.nhs.uk​/planning-and-improving-research​/policies-standards-legislation​/clinical-trials-investigational-medicinal-products-ctimps​/combined-ways-working-pilot/> accessed 5 May 2024.

58

‘Streamlining Data-Driven Research (SDDR)’ (n 55).

59

For the most pertinent, see Joshi and Morley (n 18) 64ff, which flags the contributions of the Global Digital Health Partnership, the WTO-ITU’s Focus Group on Artificial Intelligence for Health and the EQUATOR Network.

60

ibid 26.

61

ibid 27.

62

See in particular the Council’s comprehensive report, Nuffield Council on Bioethics, ‘The Collection, Linking and Use of Data in Biomedical Research and Health Care: Ethical Issues’ (2015).

63

Joshi and Morley (n 18) 27–28.

64

ibid 36–37.

65

Albeit that it is important to note that not all AI systems present the same challenge, here systems are the same: see Johan Ordish, Hannah Murfet and Alison Hall, ‘Algorithms as Medical Devices’ (PHG Foundation 2019) 23–29.

66

Joshi and Morley (n 18) 29–33.

67

Department of Health & Social Care, ‘A Guide to Good Practice for Digital and Data-Driven Health Technologies’ (GOV​.UK , 19 January 2021) <www​.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-care-technology/initial-code-of-conduct-for-data-driven-health-and-care-technology> accessed 5 May 2024.

68

ibid.

69

MHRA, ‘Good Machine Learning Practice for Medical Device Development: Guiding Principles’ (GOV​.UK , 27 October 2021) <www​.gov.uk/government/publications/good-machine-learning-practice-for-medical-device-development-guiding-principles/good-machine-learning-practice-for-medical-device-development-guiding-principles> accessed 5 May 2024.

70

Indeed, in this case, the authoring bodies explicitly identify the principles as targeting areas ‘where the International Medical Device Regulators Forum (IMDRF), international standards organizations and other collaborative bodies could work to advance GMLP’. See ibid.

71

National Strategy (n 1) 56–57.

72

Now replaced with the Data Alliance Partnership Board (DAPB). Standards DCB0129 and DCB0160 are noted in particular.

73

Harriet Unsworth and others, ‘The NICE Evidence Standards Framework for Digital Health and Care Technologies – Developing and Maintaining an Innovative Evidence Framework with Global Impact’ (2021) 7 Digital Health 1, 4; the NHS AI strategy blurs this division in suggesting that the Framework is important ‘as it is vital that those using them in the provision of care are confident that they work safely’: Joshi and Morley (n 18) 34.

74

See the predictable warning that ‘an all-too-common mistake is over-promising on the value that a product can bring’. Rebecca Boffa and others, ‘AI Regulation Guide: Considerations when Developing AI Products and Tools’ (NHS England, 9 November 2021) <https://transform​.england​.nhs.uk/ai-lab/explore-all-resources​/develop-ai/ai-regulation-guide-considerations-when-developing-ai-products-and-tools/> accessed 5 May 2024.

75

Unsworth and others (n 73) 2.

76

NICE, Evidence Standards Framework for Digital Health Technologies (2018 (updated 2022)) 22 (ESF); on the regulatory challenges of such systems, see Calvin Ho, ‘When Learning is Continuous’ in Graeme Laurie (ed), The Cambridge Handbook of Health Research Regulation (CUP 2021) 277.

77

Joshi and Morley (n 18) 34.

78

ESF (n 76) 8.

79

ibid 18.

80

ibid 44.

81

See the comments from the Association of British HealthTech Industries (ABHI) in Eliza Slawther, ‘UK’s NICE Outlines Standards that AI and Data-Driven Medtech Should Meet for NHS Uptake’ (Medtech Insight, 18 August 2022) <https://medtech.pharmaintelligence.informa.com/MT145725/UKs-NICE-Outlines-Standards-That-AI-and-Data-Driven-Medtech-Should-Meet-For-NHS-Uptake> accessed 5 May 2024.

82

See the consultation document, NICE, ‘NICE Evidence Standards Framework Update 2022 – Consultation Responses’ (2022) 1 and 10 <https://www​.nice.org​.uk/corporate/ecd7/history> accessed 5 May 2024.

83

Unsworth and others (n 73) 18.

84

ESF (n 76) 4.

85

ibid 48.

86

ESF (n 76) 26.

87

See Ada Lovelace Institute, ‘Examining the Black Box’ (2020).

88

ESF (n 76) 38.

89

Joshi and Morley (n 18) 34.

90

ibid.

91

ibid 35.

92

ESF (n 76) 4.

93

Joshi and Morley (n 18) 43, noting the efforts of the CQC, MHRA, BSI and NICE to coordinate their efforts as well as identify gaps in the overall governance ecosystem.

94

BSI and AAMI, The Emergence of Artificial Intelligence and Machine Learning Algorithms in Healthcare: Recommendations to Support Governance and Regulation (White Paper, 2019).

95

BSI and AAMI, Machine Learning AI in Medical Devices (n 49) 17.

96

Note, however, the MHRA also plays a key role in collaborating more broadly in the ecosystem, one example of which is their role in developing a range of synthetic datasets alongside the Clinical Practice Research Datalink. See Allan Tucker and others, ‘Generating High-Fidelity Synthetic Patient Data for Assessing Machine Learning Healthcare Software’ (2020) 3 NPJ Digital Medicine 1.

97

Medical Devices Regulations 2002, SI 2002/618 (as amended) reg 2(1) (MDR 2002); Optident Limited and Another v Secretary of State for Trade and Industry and Another [2001] UKHL 32, [2001] 6 WLUK 601.

98

Joshi and Morley (n 18) 22.

99

For a broader treatment of this issue, see Jean McHale, ‘Health Law, Brexit and Medical Devices: A Question of Legal Regulation and Patient Safety’ (2018) 18 Medical Law International 195.

100

Albeit without reflecting the amendments that would have been made consequent to the EU’s 2017 Medical Device Regulation.

101

MHRA, ‘Implementation of the Future Regulations’ (GOV​.UK , 28 March 2024) <https://www​.gov.uk/government​/publications​/implementation-of-the-future-regulation-of-medical-devices-and-extension-of-standstill-period​/implementation-of-the-future-regulations> accessed 5 May 2024.

102

Medical Devices (Amendment etc) (EU Exit) Regulations 2020, SI 2020/1478.

103

MHRA, ‘Regulating Medical Devices in the UK’ (GOV​.UK , 8 February 2024) <https://www​.gov.uk/guidance​/regulating-medical-devices-in-the-uk#ukcamark-and-conformity-assessment-bodies> accessed 5 May 2024.

104

MDR 2002 (n 97) regs 19, 44.

105

Christopher Hodges, ‘The Regulation of Medicinal Products and Medical Devices’ in Jean McHale and Judith Laing, Principles of Medical Law (4th edn, OUP 2017) 928. The push to recognise new UK approved bodies to carry out conformity assessments is one plausible reason for the extended transition period. DEKRA was the first such body, in late 2022.

106

ibid 928.

107

ibid 933–35; MDR 2002 (n 97) reg 7(1).

108

Medicines and Medical Devices Act 2021 (MMDA 2021) s 15(3).

109

ibid s 15(4).

110

Establishing a Pro-Innovation Approach (n 11).

111

Laura Downey and others, ‘The Medicines and Medical Devices Act 2021 & Uncertain Regulatory Futures’ (University of Birmingham, 2021) <https://www​.birmingham​.ac.uk/research/perspective​/medical-devices-act-regulation.aspx> accessed 5 May 2024; an example of the application of the framework in practice can be found in the government response to a consultation on the 2021 Act, see MHRA, ‘Medicines and Medical Devices Act 2021 Assessment’ (GOV​.UK , 26 June 2022) <https://www​.gov.uk/government​/consultations​/consultation-on-the-future-regulation-of-medical-devices-in-the-united-kingdom​/medicines-and-medical-devices-act-2021-assessment> accessed 5 May 2024.

112

Downey and others (n 111) (‘The probable consequence is that any secondary legislation made under the Act will simply add to already voluminous and complex areas of law’); see also MMDA 2021 (n 108) s 1 (the Act’s creation of a new Patient Safety Commissioner for medicines and medical devices, which may also be viewed as adding to a crowded governance ecosystem).

113

MHRA, Government Response to Consultation on the Future Regulation of Medical Devices in the United Kingdom (2022) 6 (Government Response).

114

ibid 21–22. For instance, for the latter, the UK intends to ‘broadly reflect’ the EU’s General Safety and Performance Requirements (GSPR).

115

ibid 18.

116

ibid 10.

117

ibid 8.

118

MHRA, ‘Software and AI as a Medical Device Change Programme’ (GOV​.UK , 14 June 2023) <https://www​.gov.uk/government​/publications​/software-and-ai-as-a-medical-device-change-programme> accessed 5 May 2024.

119

The same approach can be seen in the timely report of the Regulatory Horizons Council (RHC). See RHC, ‘The Regulation of Artificial Intelligence as a Medical Device’ (2022) 14 and 32.

120

Establishing a Pro-Innovation Approach (n 11) 5.

121

RHC (n 119) 4; see also Government Response (n 113) 110, noting the present regime was created before the application of AI in this area could be envisaged.

122

Set out in the guidance for the programme. MHRA, ‘Software and AI as a Medical Device Change Programme – Roadmap’ (n 37).

123

See the work of the related Software Group in acting as a focal point to bring together the various collaborating bodies. MHRA, ‘Software and Artificial Intelligence (AI) as a Medical Device’ (GOV​.UK , 3 May 2024) <https://www​.gov.uk/government​/publications​/software-and-artificial-intelligence-ai-as-a-medical-device​/software-and-artificial-intelligence-ai-as-a-medical-device> accessed 5 May 2024.

124

MHRA, ‘Software and AI as a Medical Device Change Programme – Roadmap’ (n 37).

125

Government Response (n 113) 111.

126

MHRA, ‘Medical Devices: Software Applications (Apps)’ (GOV​.UK , 1 July 2023) <https://www​.gov.uk/government​/publications​/medical-devices-software-applications-apps> accessed 5 May 2024.

127

Government Response (n 113) 114.

128

ibid 112.

129

ibid 115.

130

MHRA, ‘Software and Artificial Intelligence (AI) as a Medical Device’ (n 123).

131

Government Response (n 113) 119; see the related call from the CQC for clarity in Diagnostic Services (n 36) 11.

132

Government Response (n 113) 119. The MHRA has since clarified that this will involve secondary legislation.

133

ibid 121–22.

134

ibid 123.

135

RHC (n 119) 33–35.

136

One example could be implementing the IMDRF framework for classifying risks posed by software as a medical device, as well as for changes to essential requirements.

137

See RHC (n 119) 8, recommending that the regulatory framework for AIaMD should be ‘legislatively light’.

138

Joshi and Morley (n 18) 41.

139

A question again recently raised in the context of large language models for diagnosis, where difficulty in providing the necessary clinical evidence may also hamper regulatory scrutiny. See Johan Ordish, ‘Large Language Models and Software as a Medical Device’ (GOV​.UK , 3 March 2023) <https://medregs​.blog​.gov.uk/2023/03/03/large-language-models-and-software-as-a-medical-device> accessed 5 May 2024.

140

‘Patients Association – Patient Groups, Royal Colleges, medical charities and industry launch new Patient Coalition for AI, Data and Digital Tech in Health’ (WiredGov, 16 September 2021) <https://www​.wired-gov​.net/wg/news.nsf/articles​/Patients+Association+Patient+Groups+Royal+Colleges+medical+charities+and+industry+launch+new+Patient+Coalition+for+AI+Data+and+Digital+Tech+in+Health+16092021121500?open> accessed 5 May 2024.

141

National Strategy (n 1) 50.

142

On the latter, see a range of projects aimed at identifying and reducing such inequalities, ‘New Artificial Intelligence Projects Funded to Tackle Health Inequalities’ (NHS England, 20 October 2021) <https://transform​.england​.nhs.uk/news/new-artificial-intelligence-projects-funded-to-tackle-health-inequalities/> accessed 5 May 2024; see also the increased focus on risk inherent in the Bletchley Declaration (n 10), albeit not to the extent that the opportunities for a pro-innovation approach to regulating AI’s impact were overshadowed.

143

See Gould (n 49), highlighting the easy rhetoric of wasted opportunity in the face of regulatory timidity.

144

Roger Brownsword, ‘Regulating Automated Healthcare and Research Technologies’ in Graeme Laurie (ed), The Cambridge Handbook of Health Research Regulation (CUP 2021) 266.