Aivance Analysis | Agentic AI Governance
The comfortable assumption
Picture a fairly ordinary enterprise deployment. A research agent gathers information. A planning agent turns that information into a recommendation. A risk agent reviews the recommendation. An execution agent carries out whatever gets approved. Each of the four has been tested individually, has an approved system prompt, a defined role, scoped access, and a clean set of evaluation results.
The organisation concludes the system is safe.
That conclusion rests on a question worth asking directly: what exactly was validated, the agents, or the system they collectively create? David Manheim’s recent piece on personas and multi-agent alignment argues, convincingly, that these are not the same thing. Agent assurance does not compose. Four safe agents do not automatically add up to a safe system, and the gap between those two claims is where a large share of agentic AI risk currently lives, mostly unexamined.
From single agent to agent system
Enterprise AI architecture is shifting from a single agent calling a tool to a chain of agents that delegate to each other, consume one another’s outputs, critique or negotiate, maintain shared state, and trigger further agents downstream. The important change is not simply that there are more agents in the picture. It is that system behaviour becomes, in part, a product of interaction rather than a sum of individual behaviours. The agent stops being the natural unit of governance the moment its output depends on what another agent handed it upstream.
The compositional assurance problem
Suppose Agent A is safe, Agent B is safe, and Agent C is safe, each independently tested and cleared. It is tempting to conclude that A plus B plus C is therefore safe. That conclusion does not follow, and the reason is familiar from systems engineering generally: component assurance does not imply system assurance.
Consider a procurement agent that cannot approve purchases, a finance agent that can approve budgets, and an execution agent that can issue purchase orders. Each permission looks reasonable on its own. Chained together, the three can produce a workflow that effectively allows autonomous purchasing beyond anything governance intended, without any single agent behaving outside its stated permissions. No agent needs to be malicious or misaligned for this to happen. Incorrect assumptions passed between agents, feedback loops, cascading hallucinations, delegated authority, incomplete context, and interaction patterns nobody tested are all sufficient causes on their own.
Personas make the problem harder
Manheim’s specific contribution is to bring a persona framing to this problem. Post-training and system prompts make certain behavioural personas more likely to appear, and an aligned persona can produce consistently safe behaviour when evaluated in isolation. His concern is that alignment of a persona is conditional on the environment it operates in, an idea he draws from human role theory: people behave differently as a parent, a manager, or a competitor because behaviour is shaped by the expectations and incentives other people generate. He argues something structurally similar may hold for AI agents once they interact repeatedly with each other, so that behaviour becomes a function of model, persona, context, other agents, incentives, and interaction history combined, rather than model and persona alone.
That last step, human role dynamics generalising to AI persona dynamics and then to genuine multi-agent social structure, is the part of the argument Manheim himself treats as a prediction rather than an established fact, and he is right to be cautious about it. Human roles come with persistent identity, biological incentive, status, reputation, and long social memory. An AI agent can be reset, cloned, monitored, stripped of memory, and prevented from communicating outside defined channels, none of which apply to a person. Whether multi-agent AI systems develop anything resembling human social roles is genuinely unresolved.
The governance conclusion does not need that stronger claim to hold. Even without any social dynamic at all, interaction effects alone are enough to produce the compositional assurance problem described above. That is the more defensible version of the thesis, and the one worth building governance architecture around.
A risk category most frameworks miss
Governance thinking already accounts for model drift, where the underlying model’s behaviour changes, and data or context drift, where inputs or the operating environment shift under a system that has not itself changed. Multi-agent systems introduce a fourth category that most frameworks do not yet name.
| Drift type | What changes |
|---|---|
| Model drift | The underlying model’s behaviour |
| Data or context drift | Inputs and operating environment |
| Agent drift | An individual agent’s configuration or policy |
| Interaction drift | The relationships and dependencies between agents |
Nothing about Agent A or Agent B needs to change for the behaviour of A interacting with B to change. Memory accumulation, shifting delegation patterns, or an altered sequencing of who calls whom can gradually change what the system does in production, with every individual agent passing every individual test it was given. Monitoring agents one at a time cannot see this, because the thing that moved is the relationship, not either endpoint.
Why prompts and persona work cannot close this gap
This connects directly to the argument we made in Prompts Are Not Governance: a prompt is text the model conditions on, not a binding rule, and a well-crafted persona is a probabilistic tendency, not a guarantee. A procurement agent’s prompt can say “only execute approved transactions.” A finance agent’s persona can be built toward caution. Neither says anything about what happens once several such agents start exchanging information, context, and effectively authority with each other. If behaviour becomes increasingly dependent on context and on other agents, governance built on the assumption of behavioural stability was never going to hold, and multi-agent systems are where that assumption breaks fastest.
Moving the question: from behaviour to authority
The more useful governance question is not how do we ensure every agent behaves correctly, but what is the system allowed to cause. That reframing leads to bounded authority as the control that survives context change: rather than trying to guarantee that a procurement agent always behaves like a responsible procurement officer, an organisation defines what that agent’s identity can do, what it can do only with approval, and what it cannot do under any circumstance, independent of whatever persona happens to be active at runtime. A procurement agent might be permitted to request quotations, negotiate terms, recommend suppliers, and draft contracts; permitted with approval to commit to contracts under a defined threshold; and never permitted to modify the approved vendor list, transfer funds, approve its own contract, or change procurement policy. Persona can drift considerably inside that boundary. The boundary itself does not need to.
Decision lineage, not just action logs
A conventional audit log says an agent transferred a given sum. That is a weaker record than knowing the agent transferred that sum because a second agent recommended it, based on an analysis a third agent produced from two specific data sources, with one of four policy checks failing along the way. Multi-agent governance needs the chain, not just the endpoint: data, analysis, recommendation, delegation, approval, execution, preserved as a reconstructable sequence rather than a single logged action. That is what makes it possible to answer not just what happened, but how a set of autonomous components collectively produced it, and it is a precondition for meaningful incident response once more than one agent is involved in every material decision.
Five controls worth building toward
Taken together, the argument points to five concrete controls rather than a restatement of the problem.
| Control | What it establishes |
|---|---|
| Agent identity | Every participating agent has a persistent, attributable identity |
| Bounded authority | Authority is explicit, scoped, and independently enforceable |
| Delegation controls | Agents cannot silently transfer their authority to other agents |
| Decision lineage | Material actions preserve the chain of agents and evidence that produced them |
| Runtime admissibility | Consequential actions are evaluated against policy at execution time, regardless of what upstream agents concluded |
None of these depend on correctly predicting how any individual agent, or any individual persona, will behave once it is embedded in a system with other agents.
The governance boundary keeps moving
Governance has already moved once, from the model to the individual agent, as system prompts and evaluations shifted attention from what a model could generate to what a deployed agent could do. Manheim’s argument, read through an enterprise lens, describes the next move: from the individual agent to the system of agents and the actions that system can cause together.
We may never be able to guarantee that every agent in a dynamic multi-agent system behaves exactly as intended, and persona-level alignment work will keep improving without ever fully closing that gap. What can be defined, independent of how personas drift, is what authority those agents hold, what actions remain admissible as their behaviour changes, and what evidence has to exist before any of those actions reach the real world. That is the version of the problem an enforcement layer can actually solve.
Aivance works with CROs, CISOs, and Enterprise Architects deploying autonomous AI in Singapore and Southeast Asia. The complimentary 30-Minute Enforcement Gap Review maps where your systems have assumed authority versus where authority has been explicitly designed, including across the agents you’re chaining together. You leave with a one-page diagnosis on Aivance letterhead within 48 hours. Book your review here.