On 27 August 2026, University College London Hospitals disclosed that neurosurgeons at
the National Hospital for Neurology and Neurosurgery had removed a pituitary tumor with
real-time AI assistance. The operation took place in May. The details were held back while
the patient, 48-year-old Rhys Hibbert, recovered. He kept his sight, and he was walking
unaided within a week.
The system did not hold an instrument or move anything. It read the live endoscopic video
feed as the operation unfolded and color-coded the structures the surgeons needed to
avoid, including the carotid arteries and the optic nerves. It was developed in-house at UCL,
trained on hundreds of recorded pituitary procedures, and used under an NIHR-funded
clinical trial. The surgical team stayed in control throughout.
I want to be clear about where I stand, because the governance conversation around clinical
AI has been captured by people who treat every deployment as a hazard to be contained. I
do not see it that way. A typical UK neurosurgeon performs perhaps ten to twenty of these
procedures a year. A tool that carries pattern recognition from hundreds of prior cases into
the theater is a real expansion of what a careful surgeon can do, and the sooner it reaches
district hospitals rather than a handful of quaternary centers, the more sight gets saved.
So the question is not whether this should scale. It is what gets it there fastest. In my
experience the teams that move quickest are the ones who settled their governance
questions early, because they are the only ones who can answer a procurement panel
without going quiet.
Know Which Regime You Are Actually In
Know Which Regime You Are Actually In A lot of the commentary this week reached for US FDA language. Class II, 510(k), Software
as a Medical Device. That is the wrong map for what happened at NHNN, and using it will
cost you time.
This was a UK NHS clinical investigation. The framework is the Medical Devices
Regulations 2002 under the MHRA, with NHS layers on top: the Digital Technology
Assessment Criteria (DTAC) for procurement, the Data Security and Protection Toolkit
(DSPT) for the trust’s security posture, and the NIS Regulations 2018, under which NHS
trusts are operators of essential services. Research use sits under Health Research Authority approval and trial governance, not a commercial marketing authorization.
The practical point is about classification, and it works in your favor if you get ahead of it.
UK MDR 2002 still carries rules inherited from the old Medical Devices Directive, under
which a great deal of standalone software lands in a low risk class. The EU MDR takes a
different view. Under Rule 11, software providing information used to make diagnostic or
therapeutic decisions is Class IIa, rising to IIb or III where those decisions could cause
death or irreversible deterioration. A tool highlighting the carotid artery during a
transsphenoidal approach sits squarely in that upper territory.
The same clinical evidence can therefore produce two very different classifications
depending on the market. Teams that build to the strictest one from the start ship a single
validated product everywhere they want to sell. Teams that build to the friendliest one
rebuild their evidence package eighteen months later, usually while a customer waits.
Designing up front is the faster path, not the more cautious one.
The EU AI Act Frame Most People Are Missing
This is the part I think most of the coverage has skipped, and it is genuinely useful news for
anyone building here.
Under Article 6(1) of the EU AI Act, an AI system is high-risk where it is intended to be
used as a safety component of a product covered by the Union harmonization legislation in
Annex I, and where that product must undergo third-party conformity assessment. The
Medical Device Regulation is in Annex I. Anything above Class I under the MDR requires
notified body involvement.
That means surgical guidance software of this kind does not need a debate about its risk
tier. It arrives as high-risk by operation of law, through the device route, inheriting the full
Chapter III obligations:
- Article 9: Risk management system
- Article 10: Data and data governance
- Article 12: Automatic event logging
- Article 13: Transparency and provision of information
- Article 14: Human oversight
- Article 15: Accuracy, robustness, and cybersecurity
I count that as an advantage. The classification argument is the single biggest source of
delay I see in AI Act readiness work, and here it is pre-resolved. You know your obligations
on day one, and much of the evidence overlaps with the MDR technical file you were
building anyway. Two regimes, one body of work, if you plan them together.
Article 14 rewards a close reading. It does not simply require a human in the room. Article
14(4)(b) requires oversight measures that keep the person aware of the tendency to over-
rely on automated output. Automation bias is named directly in the statute. Design for it
deliberately and you satisfy a regulator and improve the clinical product at the same time,
because a surgeon who knows exactly when to distrust the overlay is a better user of it.
ISO/IEC 42001 is the layer that makes this survive growth. The AI Act tells you what the
system must satisfy. An AI Management System governs how the organization keeps
satisfying it through model updates, staff turnover, and expansion to new sites. That is a
scaling tool, not a paperwork exercise.
The Operating Theater Is an OT Environment, and That is Solvable
I spend most of my working life in defense contracting and CMMC, and I want to be careful
about how I use that experience here, because what transfers is control patterns, not
jurisdiction.
To be explicit: NIST SP 800-171 and CMMC do not apply to an NHS trust. Surgical video
is not Controlled Unclassified Information. It is special category personal data under Article
9 of the UK GDPR. I mention it because the CUI framing has started appearing in clinical AI
writing, and applying the wrong control set is how organizations end up with expensive
documentation that satisfies nobody.
What does transfer is the discipline of securing a processing environment you cannot take
offline. Four habits are worth borrowing:
1. Threat model for integrity and availability before confidentiality. In standard
enterprise IT the worst outcome is disclosure. In an operating theater it is a system that
fails mid-procedure, or worse, one that keeps running while quietly rendering incorrect
spatial boundaries. Confidentiality still matters. Availability and integrity are what govern
patient safety during an active case.
2. Treat the update path as the primary attack surface. A locked model prevents weight
drift at runtime, which is exactly why local deterministic inference is the right
architecture here. But locking the weights says nothing about how new weights arrive.
Demand a Software Bill of Materials, a cryptographically signed update chain, and
evidence against IEC 81001-5-1 for the health software security lifecycle. In
procurement, request a completed MDS2 and actually read it.
3. Verify your isolation posture instead of claiming it. Edge deployments get described
as air-gapped when the hardware sits on a flat clinical VLAN with a live management
interface and a vendor support tunnel. Local inference genuinely removes cloud latency
and a real class of external exposure, and that is a strong argument for deploying this
way. It is a weaker argument when the claim does not survive the first assessor who
looks. State what is actually true and the security case gets stronger.
4. Log for evidence, then reconcile with data minimization. Article 12 of the AI Act
mandates automatic recording. Article 5(1)(c) of the UK GDPR requires data
minimization. That tension is entirely resolvable, but only through deliberate retention
periods rather than defaulting to hoarding high-resolution surgical footage indefinitely.
None of this is exotic. It is ordinary OT security applied to a new setting, and it is well within
reach of any manufacturer that plans for it before the first hospital asks.
Build the Record That Lets You Defend the Product
I am not a lawyer and none of this is legal advice. But three exposures are predictable
enough to design around, and designing around them is what allows confident deployment
instead of hesitant deployment.
Clinical negligence and standard of care. Claims will center on automation bias and
whether reliance on the display was consistent with the standard of care. The answer is
protocol and documentation: clear wording that the tool is strictly advisory, a defined
procedure for handling conflicts between the surgeon and the overlay, and training records
proving the team rehearsed that exact scenario. Do this well and adoption gets easier,
because clinicians know precisely where they stand. On consent, Montgomery v Lanarkshire
Health Board governs in the UK, and material risks include the use of a novel system and
what happens to the footage afterward. One note on a citation that keeps appearing in this
discussion: the learned intermediary doctrine is a US manufacturer duty-to-warn
concept. It is not a defense to a negligence claim against a clinician, and reaching for it will
weaken an otherwise sound governance argument.
Product liability, with a date on it. Directive (EU) 2024/2853 replaces the 1985 regime
and expressly brings software, including AI systems, into scope, with member state
transposition due by 9 December 2026. It also eases the claimant’s evidential burden in
technically complex cases. The UK retains the Consumer Protection Act 1987 for now. The
takeaway is simple: the validation breadth and performance telemetry you capture today is
the same record that defends the product later. Build it once, deliberately, and it does
double duty.
Data protection and secondary use. Footage captured to treat a patient and later used to
retrain a model is a new purpose requiring its own lawful basis, and in England the national
data opt-out applies to secondary use of confidential patient information. Anonymization is
harder than it sounds when the data is high-resolution video of an individual’s anatomy, so
test that claim against ICO guidance before relying on it. Get the consent architecture right
and you unlock a continuously improving training corpus, which is the thing that makes the
next model better.
What to Put on Next Quarter’s Plan
If you are building, procuring, or advising on AI that touches a physical clinical process, five
things will move you faster:
1. Classify against the strictest market you intend to enter, and confirm early whether
Article 6(1) pulls you in through the device route. Discovering this late is the most
common cause of delay I see.
2. Design human oversight as a controlled process, with a documented disagreement
protocol and automation bias training on the record. Article 14(4)(b) is not satisfied by a
line in a user manual, and doing it properly makes the product better.
3. Put the update and maintenance path under configuration control, with an SBOM,
signed updates, and a named owner. Verify the network posture rather than describing it
aspirationally.
4. Separate the treatment lawful basis from the training lawful basis, with deliberate
retention periods and the opt-out position confirmed in writing. This is what keeps your
data pipeline open.
5. Wrap it in ISO/IEC 42001 so the controls survive the first personnel change and the
tenth deployment site.
The UCLH team did something difficult and did it inside a proper research governance
framework, which is exactly why it could be published, defended, and repeated. That is the
model. Governance did not slow this down. It is the reason there is a result to talk about at
all.
The next hundred hospitals will move at the speed of whoever has their evidence in order.