§ 01Healthcare

Healthcare

AI governance for hospitals, payers and clinical AI

Healthcare has the oldest and best-developed regulator of algorithmic decisions of any sector, and it reaches a minority of the AI actually running in a hospital. Both facts have to be held at once, and the second one is where the risk lives.

Device regulation is genuinely mature: the FDA and Health Canada have converged on the same mechanism, the predetermined change control plan, which lets a manufacturer pre-authorise how a model will be updated after it ships. That is a more sophisticated answer to model drift than any financial regulator has produced. It also only applies if your AI is a regulated device, and most clinical AI is not.

Written by Nabeel Khan, author of AI Governance & Compliance Frameworks for the Middle East · seven years in Kuwait government, the Council of Ministers and the Ministry of Planning · author of the MESA Framework · engagements are taken personally through iSystematic, not staffed to a bench
§ 02What applies

What is actually in force.

Listed by what binds you rather than by what is discussed. Every date here was verified on 9 August 2026; verify again before you rely on it, because these move.

Dec 2024
FDA predetermined change control plans, FINAL"Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions", finalised in December 2024. It is the most practically useful AI regulatory instrument in existence, because it addresses the problem every other regulator is still circling: a model that will change after authorisation. Instead of re-submitting for every update, the manufacturer specifies in advance what may change, how it will be validated and what would take it outside the plan.
Still draft
FDA lifecycle management guidance, NOT finalThe broader "Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations" was published on 6 January 2025 and remains a draft, on the FDA’s B list for finalisation in FY-2026. It covers transparency, bias, data quality, human factors, change management and cybersecurity across the total product lifecycle. Worth building to and worth describing accurately: a programme that cites it as final has misread its own evidence base.
Feb 2025
Health Canada machine-learning device guidance, FINAL"Pre-market Guidance for Machine Learning-enabled Medical Devices", finalised February 2025 after an August 2023 draft. It applies to Class II, III and IV devices using machine learning wholly or in part. Machine learning use must be explicitly disclosed in the application, a predetermined change control plan can pre-authorise planned changes, and the evidence must describe how Good Machine Learning Practice was considered and implemented across the lifecycle.
Ongoing
Health privacy, and the audit trailProvincial health privacy in Canada, where health trustees answer to PHIA and Ontario custodians to PHIPA, alongside HIPAA in the United States. For systems touching regulated records, 21 CFR Part 11 audit trails and HITRUST controls are the operational expression of it. None of this is AI-specific, and all of it applies harder to AI, because a model that reads a record at inference has touched it in a way an auditor will ask about.
The gap
Clinical AI that is not a regulated deviceTriage prompts inside an EHR, scheduling and capacity models, prior-authorisation automation, ambient documentation, coding and billing support. None of it is a medical device, none of it is inside the FDA or Health Canada pathway, and all of it affects care. There is no purpose-built framework for this category, which means the governance owner is whoever the institution appoints, and in most institutions nobody has been appointed. This is the single largest unmanaged AI exposure in healthcare today.
§ 03What does not apply

The rule that does not exist.

The instinct is to ask which regulator covers clinical AI. For most of it, the honest answer is none of them, and that is not a loophole to exploit but the actual risk position. A model that never reaches the definition of a medical device can still deny a prior authorisation, reorder a triage queue or shape what a clinician sees first, and no submission pathway will ever examine it.

Which leaves the institution to govern it, using the instruments that do exist: ISO/IEC 42001 for the management system, the NIST AI Risk Management Framework for the risk process, ISO 14971 thinking for clinical harm, and the device regulators’ own predetermined change control logic applied voluntarily to models that will never be submitted. That last one is the highest-value borrowing available, because drift is the failure mode and PCCP is the only mature answer anyone has written down.

§ 04Questions

The questions a board actually asks.

Is our clinical AI a regulated medical device?

Usually the honest answer is no, and that is the finding that reframes the programme. A device pathway is triggered by intended use, by whether the software is intended to diagnose, treat, prevent or mitigate disease, rather than by how clever the model is. Ambient documentation, scheduling, capacity forecasting, coding support and most prior-authorisation automation fall outside it. The exposure does not fall outside with them: those systems still shape care and still produce decisions a patient can be harmed by, and nobody outside the institution will review them.

What is a predetermined change control plan, and why do both regulators use it?

It is a plan, submitted and authorised in advance, describing what may change about a model after it ships, how each change will be validated, and what would fall outside the plan and require a new submission. The FDA finalised its guidance in December 2024 and Health Canada built the same mechanism into its February 2025 pre-market guidance. The convergence is the interesting part: two regulators independently concluded that the honest way to govern a system that learns is to govern its change process rather than to freeze a version. Every other sector is still trying to solve drift with periodic revalidation, and this is a better answer.

Is the FDA AI lifecycle guidance final?

No. The predetermined change control plan guidance is final, from December 2024. The broader lifecycle management and marketing submission guidance was published on 6 January 2025 and is still a draft, sitting on the FDA’s B list for finalisation during FY-2026. The distinction matters when you are writing a governance programme: a draft states the direction of travel and can change before it lands, and citing it as settled law weakens the document that cites it.

What does Health Canada require that the FDA does not?

Explicit disclosure. A Class II, III or IV application must state that machine learning is used, wholly or in part, rather than leaving it to be inferred from the technical file. The evidence must also describe how Good Machine Learning Practice was considered within the organisation and implemented across the product lifecycle, which is an organisational question rather than a product one and is answered badly by manufacturers who treat governance as documentation produced at submission time.

What is the relevant experience here?

Regulated healthcare AI, built rather than advised on. SHAP and LIME explainability dashboards for FDA-cleared clinical decision-support systems, which lifted clinician adoption by 75 per cent; a privacy-preserving federated-learning platform across more than ten hospital systems with no data sharing; a regulated retrieval system automating prior authorisation, cutting manual work by 75 per cent at 85 per cent first-pass approval with SOC 2 logging across more than a hundred HITRUST sites; and MLOps with 21 CFR Part 11 audit trails. To be precise about the boundary: those decision-support systems were FDA-cleared and the explainability layer was built for them. Securing the clearance was not the work.

Governance an examiner can follow, in the jurisdiction that actually binds you.

Fin · Healthcare
Book the 30-minute Fit Call →