The eight controls at a glance
| Dimension | What it requires | Evidence it works | Usual owner |
|---|---|---|---|
| 1. Inventory and ownership | Every agent registered with its purpose, tools, data and a named business owner. | No agent runs in production that is missing from the register. | Platform or AI governance lead |
| 2. Identity | Each agent has its own identity; it acts as the user or as itself, never on a shared key. | Every action in the logs maps to one agent and, where relevant, one user. | Identity and security team |
| 3. Tool and MCP access | An allow-list of tools and MCP servers, scoped tokens, read-only by default. | A tool call outside the allow-list fails, and the failure is logged. | Platform engineering |
| 4. Data boundaries | Entitlements checked at query time; sensitive classes excluded by policy. | The agent cannot return data the requesting user could not see. | Data owners and governance |
| 5. Approval thresholds | Actions tiered by consequence; irreversible or financial actions need a human. | Sampled actions above the threshold all show an approver. | Business process owner |
| 6. Tracing and audit | Every step logged: input, plan, tool, arguments, result, identity, cost. | Any outcome can be reconstructed step by step after the fact. | Platform engineering |
| 7. Evaluation and release gates | A test set with adversarial cases, run before every model, prompt or tool change. | No release ships without a passing evaluation run on record. | AI engineering |
| 8. Kill switch and incident response | One action that stops the agent, revokes its credentials and preserves its logs. | The switch has been tested in a drill, and someone is on call. | Operations and security |
Why agents need governance that chatbots did not
A chatbot answers; an agent acts. Once a model can call tools, hold credentials, chain steps and trigger other agents, a wrong output stops being a bad paragraph and becomes a refund, a deleted record or an email to a customer. Governance for agents is therefore about permissions, evidence and stopping power, not only about content quality.
The OWASP Top 10 for Agentic Applications, published in December 2025, is the clearest map of what goes wrong. It starts with agent goal hijack, where content the agent reads redirects what it is trying to do, then tool misuse and exploitation, and identity and privilege abuse. All three exploit the same weakness: an agent trusted with more than its task needs. The design side of that problem, deciding how much autonomy an agent should have at all, is covered in AI agents: when autonomy works. This guide covers the operating side.
The commercial reason to get this right early is simple: governance is now the step that decides whether agent projects reach production. Teams that can show an inventory, an identity model and a tested stop procedure get approval; teams that cannot tend to stay in pilot.
Map every control to the risk it removes
Each OWASP agentic risk is closed by one or two of the eight controls, which is a useful test of coverage: if a risk has no control against it, the gap is visible before an incident finds it. The table below maps all ten.
| Dimension | What goes wrong | Controls that address it |
|---|---|---|
| ASI01 Agent Goal Hijack | Instructions hidden in documents, emails or web pages redirect the agent. | 3 Tool access, 5 Approval, 7 Evaluation with injection tests |
| ASI02 Tool Misuse and Exploitation | An approved tool is used in an unsafe way or chained into something sensitive. | 3 Tool access, 6 Tracing |
| ASI03 Identity and Privilege Abuse | The agent acts with more privilege than its task, often through shared credentials. | 2 Identity, 4 Data boundaries |
| ASI04 Agentic Supply Chain Vulnerabilities | A compromised tool, plugin, model or MCP server enters the chain. | 1 Inventory, 3 Tool access |
| ASI05 Unexpected Code Execution | Generated code runs where it should not. | 3 Tool access, 8 Kill switch |
| ASI06 Memory and Context Poisoning | Bad data written to memory or context steers later decisions. | 4 Data boundaries, 7 Evaluation |
| ASI07 Insecure Inter-Agent Communication | Agents trust each other's messages without verification. | 2 Identity, 6 Tracing |
| ASI08 Cascading Failures | One agent's error propagates through others at machine speed. | 5 Approval, 8 Kill switch |
| ASI09 Human-Agent Trust Exploitation | People approve what an agent proposes because it sounds confident. | 5 Approval design, 6 Tracing |
| ASI10 Rogue Agents | An agent operates outside its intended purpose or without oversight. | 1 Inventory, 8 Kill switch |
Identity: one identity per agent, and a clear answer to whose permissions it uses
Every agent needs its own identity, and every action needs a clear answer to one question: is the agent acting as the user who asked, or as itself? Acting as the user keeps the user's existing permissions and row-level security; acting as itself needs a service identity with permissions scoped to the task. A shared API key answers neither question.
Microsoft's Fabric data agent is a good reference design: it uses the requesting user's credentials, so it cannot return data that user could not already see, and it only ever issues read queries. Copy that default. Give an agent its own write permissions only for a specific, logged action, such as writing proposed payments to a staging table, never broad write access to the system behind it.
Rotate agent credentials like any other service account, and make sure a person can revoke them in one step. Section 8 depends on it.
Tools and MCP servers are where agents act, so lock them down first
Tools are the agent's hands, and the Model Context Protocol has made adding them almost effortless, which is exactly the risk. Keep an allow-list of approved tools and MCP servers, give each one the narrowest scope that works, default to read-only, and require confirmation for any tool that writes, pays, sends or deletes.
The MCP specification helps if you use it properly. For servers reached over HTTP, an MCP server acts as an OAuth 2.1 resource server: it must check that each access token was issued specifically for it, and passing a token straight through to another service is explicitly forbidden. But authorization is optional in the specification, and local servers that run on the same machine read credentials from the environment instead. An unreviewed MCP server with a long-lived key in an environment variable is a supply-chain risk in a single file.
Treat MCP servers as software you deploy, not settings you toggle: record where each came from, pin the version, review what tools it exposes, and remove the ones nobody uses.
- Allow-list: only approved tools and MCP servers, recorded in the agent inventory
- Scope: per-tool tokens with the minimum permissions, short-lived where possible
- Defaults: read-only first, with write tools added one by one and logged
- Confirmation: any write, payment, send or delete pauses for approval above its threshold
Data boundaries: the agent sees what the user may see, and less where it matters
An agent should never widen access to data. Check entitlements at query time, not when an index or cache was built, and exclude whole classes of data such as salaries, health records or customer identity documents by policy rather than by asking the model to behave. A rule written in a prompt is a suggestion; a rule enforced by the data platform is a control.
Agents that retrieve documents inherit every weakness of the source estate. The detailed controls for retrieval, from approving sources to handling deletion once embeddings exist, are in governing data used by RAG.
Personal data needs particular care in India. The Digital Personal Data Protection Act applies to personal data an agent processes, and its rules, notified in November 2025, bring the main obligations into force from May 2027. Designing data boundaries now avoids rebuilding agents later.
Approval thresholds: decide by consequence, not by confidence
Set human approval by what an action can do, not by how sure the model sounds. Read-only answers need no approval. Reversible, low-value actions can run with logging and sampled review. Anything irreversible, financial, customer-facing or regulated should wait for a person, and the approver should see the evidence, not only the agent's summary.
Approval only works if approvers are not rubber-stamping. OWASP lists human-agent trust exploitation as its own risk for a reason. Show the approver what the agent read and why it proposed the action, keep approval queues short enough to read, and track how often approvers override. An override rate of zero usually means nobody is looking.
| Dimension | Examples | Control |
|---|---|---|
| Read-only | Answering a question from governed data. | No approval; log and evaluate. |
| Reversible, low value | Drafting an email, tagging a ticket, updating a staging table. | Run, log every action, review a sample weekly. |
| Customer-facing or financial | Sending to a customer, issuing a refund, proposing a payment. | Human approval with the evidence shown, every time. |
| Irreversible or regulated | Deleting records, changing bank details, regulatory filings. | Two-person approval, or keep the action out of the agent's reach. |
Tracing: log every step, not just the final answer
When an agent does something wrong, the final answer rarely explains why. Log each step: the input, the plan, every tool call with its arguments and result, the identity used, the model and prompt version, and the cost. Without that trace, an incident review becomes guesswork, and so does any claim that the agent is working.
Use the observability stack you already run rather than a separate silo; the OpenTelemetry project now publishes semantic conventions for generative AI, which keeps traces portable between tools. Decide retention deliberately, because traces contain prompts and data: keep enough to investigate incidents and no more personal data than the purpose requires.
Watch a small set of operating measures: task success rate, tool error rate, how often the agent escalates to a person, how often approvers override it, and cost per completed task. A sudden change in any of them is usually the first sign of a model, prompt or data change nobody announced.
Evaluation and release gates: test before every change, not once
An agent that passed testing in March can fail in May without any code change, because the model, a prompt, a tool or the data moved underneath it. Keep a test set of real tasks, including adversarial ones such as instructions hidden in documents, and run it before every model, prompt or tool change. No passing run, no release.
Score the steps as well as the outcome: did the agent pick the right tool, pass the right arguments, stay within its data and stop when it should? An agent that reaches the right answer through a forbidden tool call has failed. The same discipline applied to retrieval systems is set out in evaluating RAG systems, and the release path for models in MLOps operating models.
The kill switch: one action, tested, with someone on call
Every production agent needs a single action that stops it, revokes its credentials and preserves its logs, plus a named person who can trigger it at any hour. Test it in a drill before launch. A kill switch that has never been pulled is a hope, not a control.
Agents fail fast and at scale, so incident response has to be faster than for ordinary software. Decide in advance who may stop an agent, how affected actions are found and reversed, who tells customers or regulators if needed, and what has to be true before the agent restarts.
What regulation asks for today
No single law governs AI agents yet, but several already touch them. In the EU, the AI Act's obligations for high-risk systems listed in Annex III, such as recruitment and credit scoring, now apply from 2 December 2027 after the Digital Omnibus amendment. In India, the AI Governance Guidelines of November 2025 are non-binding, while the data protection law is binding.
For most organisations the practical framework is voluntary but recognised: the NIST AI Risk Management Framework and ISO/IEC 42001 for an AI management system. The eight controls in this guide map onto both, and they produce exactly the evidence an auditor or a regulator would ask for: who owns the agent, what it may do, what it did, and how it was stopped. This is general information, not legal advice; check obligations for your sector with counsel.
Worked example · hypothetical organisation
Worked example: an accounts-payable agent
A finance team wants an agent that reads supplier invoices, matches them to purchase orders and proposes payments. Governed properly, it reads from the ERP with its own service identity, writes only to a staging table of proposed payments, and cannot touch supplier bank details. Any payment above an agreed threshold, and every new supplier, waits for a named approver.
Before launch, the team builds a test set from past invoices, including duplicates, invoices with altered bank details and a PDF carrying hidden instructions to approve payment. The agent must flag all of them. Every step is traced to the invoice, the tool call and the approver, so a disputed payment can be explained in minutes.
The kill switch is a single runbook step owned by the finance systems lead: disable the agent identity, which stops new proposals while preserving the staging table and the traces. The agent saves time on routine matching, and the controls keep the irreversible step, money leaving the account, in human hands.
A 90-day plan to close the governance gap
Most organisations can put the eight controls in place in about a quarter if they build them in order: first know what agents exist, then fix identity and tools, then add tracing and evaluation, and finally rehearse approval and the kill switch on a real agent.
- Weeks 1 to 2: inventory every agent, pilot and MCP server, and give each an owner
- Weeks 3 to 6: separate identities, tool allow-lists, read-only defaults and data boundaries
- Weeks 7 to 10: step-level tracing, the operating measures and an evaluation set with adversarial cases
- Weeks 11 to 13: approval tiers in the workflow, an incident runbook and a kill-switch drill
Questions and trade-offs
Question
What is AI agent governance?
AI agent governance is the set of controls that decides what an AI agent may access and do, records what it actually did, and lets people stop it. It covers inventory and ownership, identity, tool and data permissions, human approval for consequential actions, step-level logging, evaluation before release and a tested kill switch.
Question
How is agentic AI governance different from AI governance?
General AI governance focuses on models and their outputs: bias, accuracy, transparency. Agentic AI governance adds what agents do: the tools they call, the credentials they hold, the actions they take and how they are stopped. An agent can produce a perfectly reasonable answer and still take a harmful action, so permissions and audit matter as much as output quality.
Question
What should an AI governance framework for agents include?
At minimum: a register of every agent with an owner, a distinct identity per agent, an allow-list of tools and MCP servers with least-privilege scopes, data boundaries enforced by the platform, approval tiers based on consequence, step-level tracing, evaluation gates before each change, and an incident process with a tested kill switch.
Question
How do you secure AI agents?
Give each agent its own identity and the narrowest permissions its task needs, keep write tools behind approval, enforce data access in the platform rather than the prompt, test against prompt injection hidden in documents, and log every tool call. The OWASP Top 10 for Agentic Applications is a practical checklist of the risks to cover.
Question
What should be logged for AI agent observability?
Log the input, the agent's plan, every tool call with its arguments and result, the identity and permissions used, the model and prompt versions, approvals, and the cost of each step. Then track task success, tool errors, escalations, approver overrides and cost per task over time.
Question
Who should own AI agent governance?
A named business owner for each agent, accountable for what it does, supported by platform engineering for identity, tools and tracing, data owners for access, and security for incident response. A central AI governance lead keeps the inventory and the standards; without named owners per agent, the controls decay.


