Agentic AI — systems that take sequences of actions autonomously, making decisions at each step rather than returning a single output — is arriving in regulated industries. The pilot programmes are running. The production cases are starting to emerge. The governance frameworks are not ready.
This is not a criticism. The technology has moved faster than the regulatory thinking in most jurisdictions. Regulators who were still working out how to treat a single AI decision are now being asked to think about AI systems that make dozens of decisions in sequence, in real time, with compounding effects that are difficult to audit after the fact.
The challenge for organisations deploying agentic systems in regulated contexts is that the existing compliance architecture was designed for a different class of system. Audit trails assume discrete decision points. Explainability requirements assume a single output with traceable inputs. Human oversight assumes a human has enough time to review what the system is doing before it does it.
None of those assumptions hold cleanly for a well-designed agentic system, which is precisely what makes it useful — and precisely what makes it hard to govern.
What makes agentic systems different from standard AI outputs
A standard AI integration produces a result. A document is classified. A risk score is assigned. A recommendation is generated. The human decides what to do with it. The audit trail is relatively straightforward: input, model version, output, human decision.
An agentic system takes action. It searches, retrieves, decides, calls an API, updates a record, escalates, summarises, and routes — potentially all within a single triggered workflow. Each step depends on the output of the previous one, which means the chain of reasoning can compound in ways that are difficult to reconstruct after the fact.
For a compliance team trying to explain what the system did and why, this creates a genuine problem. The answer is not a single output. It is a sequence of intermediate states, each of which may have been influenced by data that is no longer in scope.
The governance patterns that do work
The organisations getting this right are building governance in from the design stage, not adding it retrospectively. Several patterns are emerging as reliable:
Explicit action boundaries. Define, at design time, what the agent is and is not permitted to do. Not as a policy document — as a technical constraint enforced in the implementation. An agent that cannot update a customer record without a human-reviewed intermediate state is not a limitation; it is a governance feature.
Structured audit trails at every step. Each action in the agentic chain should produce a log entry that records the state before the action, the reasoning behind it, the action taken, and the state after. This is more data than a standard system produces, but it is the minimum required to reconstruct what happened in a regulatory enquiry.
Human-in-the-loop thresholds. Define in advance the conditions under which the system must pause and seek human review before proceeding. These thresholds should be based on consequence, not just probability. A low-probability action with significant downstream effects should trigger review even if the agent's confidence is high.
Rollback capability. Some actions are reversible and some are not. Agentic systems operating in regulated environments should prefer reversible actions where possible, and should be explicit in their design about which actions cross the irreversibility threshold.
The regulatory direction of travel
The FCA, PRA, and their international equivalents are all moving towards requiring demonstrable human oversight of consequential AI decisions — not just documented policy that oversight exists. The SS1/23 supervisory statement on model risk management, and the emerging AI governance frameworks building on it, point in a consistent direction: the burden of proof is on the organisation to show that governance actually governed.
For agentic systems, this means the audit trail is not optional and not a retrospective exercise. It needs to be designed into the system from the outset, because retrofitting it after the fact is considerably harder than building it in.
The organisations that will deploy agentic AI at scale in regulated environments are the ones treating governance as a first-class technical requirement — not as a compliance task to be handled by a separate team after the engineers have finished.
If you are designing agentic systems for a regulated context and want to think through the governance architecture, our AI Services approach sets out how we approach this from day one.