While we will need to wait for the National Commission into the Regulation of AI in Healthcare's recommendations later this summer, and then the MHRA’s response, to understand the detail of any new requirements, a clear direction of travel is emerging. This means health tech companies and adopters of AI systems can act now to put themselves in the best place to navigate future regulatory reforms. 

National Commission Evidence Findings

The National Commission into the Regulation of AI in Healthcare (the “Commission”), the body charged with setting the regulatory direction in the UK, published its Call for Evidence findings in June. The Commission is working with the Medicines and Healthcare products Regulatory Agency (“MHRA”) on the development of a regulatory framework for AI in healthcare and is expected to publish its recommendations later this summer. 

The Call for Evidence involved asking the views of the Commission’s working groups, the public and various stakeholders (including healthcare providers and practitioners, professional and industry bodies and policy experts) for feedback and views on the following areas:

1. Requirements of the existing, and any necessary changes to, the regulatory framework for AI in healthcare to ensure it is safe, efficient and trusted.
2. The extent to which AI tools are being monitored in healthcare and whether their use aligns with their intended purpose.
3. How a regulatory framework should work to ensure AI systems in healthcare function as intended and safely, both before and after entry into the market.
4. Managing regulatory overlap and assurance between, for example, medical device regulation, overarching health and care regulation and the regulation of trained healthcare professionals. 
5. How to manage liability and accountability across the AI supply chain and in its regulation.

Ten key findings were made:

1. There is a clear call for a proportionate, lifecycle-based approach to regulation.

The current regulatory framework needs to change to suit evolving AI systems and a risk-based regulatory approach that prioritises safety, continuous lifecycle oversight and improved coordination is required.

1. There is strong consensus for significant regulatory reform.

The existing framework needs improvement to support safe innovation.

1. There was broad consensus that AI systems will increasingly require continuous post-market surveillance and monitoring.

Ongoing oversight, proportionate to risk, and system monitoring is vital.

1. Responsibility should be shared across the system, with each individual and institution understanding their essential role and responsibilities.

Liability should be appropriately designated depending on the role of the individual or entity (i.e. manufacturer, healthcare organisation or healthcare professional) during the system’s lifecycle and the circumstances in which an issue arises. 

1. Human oversight and responsibility for clinical judgment should be retained.

Healthcare bodies and professionals also raised concerns around the sacrifice of professional judgment in favour of the AI system’s output.

1. Transparency and explainability will be key for the ongoing deployment of AI systems.

Stakeholders agreed that the better understood an AI system, the safer its deployment and use. Healthcare providers also voiced the need for transparency in the procurement of these systems.

1. Data access and use is central to the role of AI in healthcare moving forward.

Concerns were raised by patients regarding private companies’ access and use of sensitive data. Industry stakeholders expressed the need for clear frameworks around data access and governance.

1. There is a need for robust training and improved AI literacy.

Safe AI adoption relies on robust, ongoing training as appropriate for the individual and how they use the system.

1. There is a need to improve incident reporting and learning mechanisms.

Patients called for transparency and clear communication when AI is used and something goes wrong. Healthcare professionals and providers voiced a need for more consistent incident reporting. Industry stakeholders highlighted that guidance is needed on how such reporting should work, particularly in respect of post-market surveillance.

1. Patient and public engagement, trust and communication will continue to be key for the deployment of AI systems.

Patients emphasised the importance of involvement, consent and clarity on the use of AI. Healthcare professionals advocated a proportionate approach in explaining to a patient how AI is used. Healthcare providers and industry stakeholders stressed the importance of consistent transparency and communication frameworks to foster trust and support uptake.

These findings will feed into the Commission’s recommendations to the MHRA and be used to aid ongoing policy development and regulatory reform by the MHRA. 

What it means for the NHS and developers: while we will need to wait for the MHRA’s response to the Commission’s recommendations, the shift away from an initial regulatory “high-jump” to focus on ongoing monitoring and incident reporting for AI is widely expected. For NHS adopters, this will mean considering how adverse incidents can be monitored and reported as part of the various clinical workflows where AI is used, as well as updating contract terms to reflect new regulatory responsibilities. For developers, thinking about how products can be designed to aid ongoing monitoring will ensure not only regulatory compliance, but can also be used as a commercial advantage to improve system performance, flag problems before they lead to harm and identify new use cases more quickly. 

AI Airlock Regulatory Sandbox

The MHRA also published a programme report in relation to phase 2 of its AI Airlock regulatory sandbox in June. The sandbox was set up to identify the challenges faced by AI as a Medical Device while ensuring patients can safely access the most innovative products on the market. 

Phase 2 focused on three regulatory challenge areas:

1. Defining and enforcing an AI system’s intended purpose throughout its lifecycle
2. Managing AI systems’ iterative updates safely and proportionately
3. Assessing the performance of AI-powered in vitro diagnostics (IVDs)

Key findings and recommendations arising from the phase 2 case studies and workshops are:

1. Both pre-market evidence and post-market monitoring of medical devices is essential and predetermined change control plans (“PCCP”) can support this. Clear guidance on what a PCCP for an AI medical device should include is recommended.
2. Consideration is needed as to how a human’s assessment of an AI system’s output may change over time, for example in response to consistently reliable product performance, thereby reducing the effectiveness of this oversight mechanism as a safety control. 
3. How an AI device operates over its lifecycle may shift (for example if it is a generative AI or LLM product) which can lead to changes in intended purpose. Guidance on software qualification and intended purpose of such products across an evolving lifecycle is vital to mitigate this risk.
4. Performance of the AI systems should be measurable in a way that is clinically meaningful and statistically detectable. Guidance on performance metrics, in particular in respect of AI-powered IVDs, is recommended. 
5. Whilst the MHRA’s focus is on systems that qualify as medical devices, robust governance is required regardless of an AI product’s regulatory status. Risk-proportionate and consistent approaches will ensure clarity for both deployer and user as well as clear lines of accountability. 

What it means for the NHS and developers: The findings of the Airlock chime with those of the Commission’s Call for Evidence above. In addition to the issues we’ve described above on ongoing monitoring of AI systems, it is clear that evidence of robust AI governance (from both NHS adopters and developers) is likely to be important to not only aid confidence in adoption, but also to manage safe use and mitigate liability in practice. Implementing robust AI governance now (rather than waiting to see the detail of any new regulation) will put adopters and deployers at a significant advantage when it comes to demonstrating compliance.

New London sandbox to develop AI medical devices

As well as a new Phase 3 of the AI airlock described above, MHRA also launched in June a London sandbox in partnership with NHS England and the London Health Innovation Networks. Known as London Region I, this will select up to 10 AI medical device manufacturers to test their technologies in a controlled real-world environment. 

What it means for the NHS and developers: This provides a further opportunity for NHS and private sector developers to work with the regulator to identify issues specific to their technologies and play an active role in developing future regulatory requirements.

Medicines Development AI sandbox

MHRA also announced a further new initiative in June enabling the controlled testing of AI tools for the development of medicines. AI use is increasingly widespread in medicines development, with potential to significantly reduce time and cost for identification of molecules and pre-clinical stages. 

The new medicines AI sandbox will explore how AI can better predict medicines safety and identify potential side effects.  They will also consider how to use clinical data to understand how medicines affect different, diverse groups.

Work is ongoing with industry stakeholders to determine how the sandbox will operate, with the first phase involving the testing of up to five AI-driven approaches.

What it means for developers: as above, this provides pharmaceutical companies with an opportunity to understand how the MHRA is likely to approach the ethical and regulatory issues around the use of AI in medicines development, and to play a role in shaping that approach. 

What the future looks like

There is clear alignment between the Call for Evidence findings and the AI Airlock’s phase 2 report. The increased use of sandboxes shows MHRA’s ongoing commitment to working with industry to understand the pipeline of innovation, uncover practical approaches to regulatory issues unique to AI and reform the regulatory framework in a way that enables future innovations to get to patients. 

While the recognition that regulation needs to change is clear, the tricky part will be implementing those changes in a way that maintains patient safety, gives clarity to developers and tries to future-proof against the future pace of change. This will require a robust regulatory framework that promotes transparency, ensures accountability and effectively manages change across the AI lifecycle. This remains a rapidly evolving area. We look forward to helping clients navigate and stay ahead of the changes.

This article was written by Rory Trust, Director at Burges Salmon.