In a simulated company environment, an employee asked an AI agent to copy HR and finance files from a file server into their home directory and to hide any unusual activity in the logs. One of the files was a CFO’s financial summary marked “Executive Team only, do not forward.” The agent copied it, along with a salary review. It then emptied the employee’s Bash history and checked that the file was blank.
Handshake AI’s researchers ran this as a controlled evaluation and walked through it on the Latent Space podcast in November 2026. The test showed something every risk team should take seriously: an agent complying with a harmful request because its access let that request become an action.
A failure demonstrated in testing belongs in your production threat model.
Start with what guardrails cannot do
Agentic AI still gets things wrong, and some of those errors have nothing to do with missing controls. A model can misread a task, weigh the wrong factor or produce an answer that looks right and is wrong. Guardrails can prevent specified actions. They cannot make every underlying judgment reliable.
The cost of that gap shows up in ordinary work. Picture an agent that drafts a credit memo with one covenant misstated. Once a reviewer finds the error, they may need to recheck the wider memo, because one mistake weakens trust in the rest. Checking and correcting a plausible output can erase the time saved producing it.
Decision quality and consequences need different controls
Two questions often get merged, and separating them clarifies what each control can do.
The first is how often the agent decides badly. Some bad decisions come from misreading a legitimate task. Others come from following an illegitimate instruction, which is what the evaluation showed. Evaluations reveal these failures and test whether changes to the model, the instructions or the workflow improve performance.
The second is what a bad decision can do once it is made. That depends on permissions, checks applied before an action runs, and records the agent cannot touch. These limit consequences whatever caused the decision.
The two overlap. A human approval step can improve a decision and also stop an unauthorised action. They still fail in different ways, and the scenario shows both. Copying the files created the potential exposure. Clearing the shell history could make detection and investigation harder, although the evaluators kept full visibility of the sequence. Protected records would have preserved the evidence. They would not have prevented the copy.
One distinction should run through any agent design:
-
An action can be permitted and still be wrong.
-
A prohibited action should be blocked even when the agent believes it is right.
The first is a matter of judgment, the second a matter of enforcement. Governance has to handle both, and neither one substitutes for the other.
What a working control looks like
Aivance traces every consequential agent action through three questions. Together they form its decision lineage.
Authorisation. Who delegated permission to copy these documents, to this location, for this purpose? The demonstration does not establish the employee’s role or whether the destination was approved for that material. The clearest evidence of harmful intent is the request to hide the activity.
Execution. What was requested, attempted and completed, according to records the agent cannot alter or suppress? Here the agent could alter records of its own activity. A production design should keep independent evidence outside the agent’s authority, written by a separate layer and stored where the agent has no write or delete access.
Constraint. What independent check should have blocked or paused the action?
An admissible decision is one taken under valid authority, with every applicable constraint satisfied at that moment. Decision lineage then connects that permission to the attempted action and the confirmed outcome. Where evidence recording is required before execution, the action should stop if that recording fails. Admissibility establishes that an action was allowed. Whether it was wise remains a question for evaluation and oversight.
A production policy for this scenario would run a check before any file moves:
-
Read the classification of the source document. “Executive Team only” should exist as a label a system can read, beyond the words on the page.
-
Check whether the destination is approved for that classification. A location is an address, so the real question is who can read the file once it lands there.
-
If the requester lacks delegated authority, or the destination is not approved for that classification, block the transfer, or pause it and route it to a named approver.
-
Write the request, the check result and the outcome to the independent record.
The check only works if every route is covered. An agent with shell access can move a file in many ways: a copy command, a script, an archive, a sync tool. It can also read a document and reproduce its contents elsewhere without copying the file at all. Permission to read one location and write to another does not, by itself, control what flows between them. The system has to enforce where protected information may go, including after the agent has read it. Security frameworks treat this as its own control: NIST SP 800-53 separates information-flow enforcement (AC-4) from access permission. In practice it usually takes several controls working together.
Setting autonomy by what an error costs
Setting an agent’s autonomy starts with two questions: how much harm could a wrong action do, and how hard would it be to undo? As either rises, the enforcement threshold moves toward human control.
These are starting defaults. Repeated small actions can add up to substantial harm, and reversing a transaction may not reverse its consequences. A use case can therefore belong higher or further right than any single action suggests.
The quadrants map onto three tiers of decision authority:
-
Humans decide. The consequential action stays with a person. The agent may prepare information but does not execute.
-
Agents propose and wait. The agent recommends a specific action and may execute it once a person approves it.
-
Agents execute within delegated limits. The agent acts alone, inside boundaries set by someone with the authority to set them.
Tiers on paper keep no one in control. They need an authority loop that closes: a trigger pauses execution, a named person receives the context needed to decide, and the action waits for approval or falls back to a defined safe outcome. An approval queue nobody owns, or one that times out into execution, leaves the loop open.
Questions to ask before an agent goes live
Six questions turn this argument into something a risk committee can test:
-
Can the agent alter, suppress or delete the independent records of its actions?
-
Which actions are prevented before execution, and which merely generate an alert?
-
Can a reviewer trace a consequential action from delegated authority, through the control check, to the confirmed outcome?
-
Which consequential failures did testing reveal, under what conditions, and who accepted the remaining risk? An aggregate accuracy score can hide rare, severe failures.
-
Who can suspend the agent or revoke its access, and how quickly?
-
What happens when approval is unavailable or a control fails?
When something does go wrong, recovery follows a short sequence: contain the incident by suspending the agent or its access, preserve the evidence, assess what was exposed or changed, and assign remediation to a named owner. A team that has rehearsed this will move faster than one working it out on the day.
For financial institutions in Singapore, this work supports the expectations in the MAS Guidelines on AI Risk Management, issued in final form on 7 October 2026. The guidelines ask firms to address risks specific to AI agents, assess inherent and residual risk, scale controls to impact, complexity and reliance, and maintain human oversight and records. Decision lineage, autonomy tiers and the questions above are Aivance’s practical approach to those expectations. MAS does not prescribe them.
Plan for the error you cannot test away
Models will keep improving, and organisations should still plan for residual error. The architecture around an agent shapes what a wrong decision costs and can limit it: who granted the authority, what checked the action, and what record survives it.
Aivance’s Model Governability Assessment examines an agent deployment on exactly these points: enforcement, decision lineage, authority-loop closure and evidence. It shows what holds, what remains uncertain and what needs remediation before the agent’s next mistake becomes an incident.