← All notes
AI oversight

An AI assistant that can spend money needs a permission someone can see

Escherbridge

The usual AI assistant demo has a plot. The assistant gets a goal, uses a few tools, and comes back with a result and an air of competence. The missing scene is the boring one: who allowed it to use those tools, how far the permission went, whether anyone could have stopped it halfway, and where a manager can read all of that back on a Tuesday.

Without that scene, “autonomous” is a synonym for “we will reconstruct it from logs if something expensive happens.”

The regulation is about the log, not the vibe

Article 12 of the EU Artificial Intelligence Act, Regulation (EU) 2024/1689, is the record-keeping article for high-risk systems. It requires those systems to technically allow automatic recording of events over the lifetime of the system, at a level that supports tracing what the system did. It is not a suggestion that a vendor keep some debug output somewhere. It is a design requirement, aimed at systems whose failures are not aesthetic. The rest of the Act still sorts systems by risk, and a chatbot that recommends a lunch is not a high-risk credit model. The useful idea travels anyway: if the system can take an action that matters, the action should be reconstructable. EUR-Lex, Regulation (EU) 2024/1689.

NIST’s AI Risk Management Framework, published in 2023 and widely used as a voluntary map, sorts the work into Govern, Map, Measure, and Manage. Govern is the unfashionable function: roles, accountability, and the policy that says who owns an outcome. The playbook’s Govern material keeps returning to the same demand — name the human roles, and do not leave oversight as an atmosphere. A framework is not a fine. It is a way to notice that an agent with tools and no named owner is an accountability hole you have scheduled for later. NIST AI RMF, Playbook (PDF).

Consent is a grant, or it is a checkbox

  • A terms-of-service click does not say which tool the agent may call, against whose data, until when.
  • A chat transcript is not an authorization. It is evidence that a conversation happened.
  • A vendor debug log that the subject cannot see is a log for the vendor.
  • Revocation that takes effect “on the next deploy” is not revocation. It is a queue.

The practical test is small. Can a person point at the grant, narrow it, or kill it, and can a second person see that this happened without asking the model to recall? If the answer is no, the agent is acting on manners. Manners are not a control.

The habit, and a better one

The habit is demo-ware: an agent that finishes the task in the video and cannot show the permission in the log. The constructive fix is to separate the draft from the act. Let the model propose. Let a grant, with a named person and an expiry, be what allows the tool to run. On our own engine this is not a slogan. In October 2026 the tenant work landed a plain rule: a claimed member keeps working under their own grant (ffde102), and the follow-up refuses to mint wallets into claimed accounts or to describe consent as broader than it is (da1a4b6). Real-value settlement is still switched off. The permission is not. That is the order we want the industry to copy.

Sources

  1. Regulation (EU) 2024/1689, the Artificial Intelligence Act (EUR-Lex)
  2. NIST AI Risk Management Framework
  3. NIST AI RMF Playbook