Policy Simulation: terraform plan for AI governance

A security rule that nobody dares switch on protects exactly nothing. The blocker is never the rule — it is not knowing what it will break.

Here is a pattern anyone who has rolled out a governance control will recognise. You write the rule. You are confident it is correct. You set it to monitor-only “just for a week” to see what it catches. A month later it is still in monitor-only, because nobody can say with confidence what flipping it to enforcing would do to production traffic — and the cost of being wrong is an outage, while the cost of waiting is invisible.

So the control sits there, logging. It has all the operational cost of a security boundary and none of the protection.

The gap was never a better policy engine. ElsiumAI already has several — the policy engine, capability tokens, information-flow control. The gap was an answer to a single question: what will this block?

The answer was already on disk

Every agent run in ElsiumAI can produce a signed ExecutionProof: a hash-chained record of the LLM calls, tool calls, retrievals and policy evaluations that made up that run. They were built for after-the-fact verification — proving what happened.

But a corpus of proofs is also a corpus of decision points. Replay a candidate policy over them and you are no longer guessing at the blast radius; you are measuring it against traffic that actually occurred.

import { simulatePolicy, flowPolicyProbe, formatSimulation } from '@elsium-ai/observe'
import { createFlowPolicy, createLabel, lethalTrifectaRule } from '@elsium-ai/core'

const result = simulatePolicy(
  traces,                                    // 100 recorded runs
  flowPolicyProbe({
    policy: createFlowPolicy([lethalTrifectaRule()]),
    initial: createLabel({ classes: ['secret'], origin: 'trusted', source: 'vault' }),
  }),
)

console.log(formatSimulation(result))

Same shape as terraform plan or opa eval: decide against history, read the plan, then commit.

The example that motivated the feature

A corpus of 100 runs. Ninety-seven ordinary. In three of them a retrieved document was attacker-controlled and the agent subsequently reached for an outbound tool — the lethal trifecta: sensitive data, untrusted content, and a way out.

Run lethalTrifectaRule() with its defaults over that corpus:

200 decision point(s) across 100 run(s): 100 allowed, 100 denied
100 of 100 run(s) affected (100.00%)

Every single run. And the rule is not wrong — it is doing precisely what it says. Its default sink list covers tool:*, so once a secret and untrusted content are both in the context, it stops the agent reaching for any tool, including read-only ones that could not exfiltrate anything.

Shipping that on a Friday takes production down. Now narrow the sinks to genuine egress:

200 decision point(s) across 100 run(s): 197 allowed, 3 denied
3 of 100 run(s) affected (3.00%)
  ✗ proof_… @2 tool:send_email

Three runs. The three that were actually attacks. And you can open each one by traceId.

Both versions of that rule are defensible on paper. Only one is shippable, and the difference is a single option that no amount of reading the rule would have surfaced.

The two numbers worth arguing about

affectedTraceRatio counts runs touched, not decision points. That distinction is deliberate: one run breaking is what a user experiences. A rule that fires twice within the same run is one incident, not two, and a percentage computed over decision points would quietly overstate the damage.

newlyAllowed exists because governance erodes in the direction nobody watches. When you compare two policies, everyone asks for the blast radius:

const plan = comparePolicies(traces, { baseline: current, candidate: proposed })
// { newlyDenied: [...], newlyAllowed: [...], unchanged: 197, ... }

newlyDenied is what people request. newlyAllowed is what they forget to request — a relaxed policy silently permitting things it used to block is how a control decays into decoration, one reasonable-looking exception at a time.

What makes the plan trustworthy

• It is read-only. Nothing is executed and no side effect is replayed. A tool call denied in simulation still happened in the recording — that is the finding, not a bug.
• The corpus is tamper-evident. Proofs are hash-chained and signed, so nobody can edit history into a friendlier plan.
• classify is a hook, not an omission. Proofs store hashes rather than content by design, so a proof can be shared with an auditor without leaking what it processed. Sensitivity classes therefore have to come from the operator, and the API says so rather than pretending to infer them.
• Ordering is deliberate. A tool is checked before its own output joins the context. Otherwise every tool would appear to taint itself and every simulation would return noise. There is a test for exactly that.

One more design note. PolicyProbe is a port, not a closed set:

type PolicyProbe = (trace: ExecutionProof) => SimulatedDecision[]

Policy engines differ in what they decide over — flows to a sink, authorization requests, budgets — and the simulator should not need to know. flowPolicyProbe ships with the package; a capability-token probe or your own engine plugs in the same way.

The part where the tests were right and I was wrong

Four tests failed the first time this ran, on a fixture I had written to be obviously benign. The fixture was a normal run calling lookup_order, and the simulator denied it.

The code was correct and the fixture was wrong. tool:* really does cover lookup_order. My own intuition about what the default rule would block was off — on a corpus I had constructed myself, for a feature I was building specifically to answer that question.

That failure is now a test in its own right. If the author of the rule cannot predict its blast radius on a hand-made fixture, expecting anyone to predict it on production traffic was never realistic.

Try it

The runnable corpus is in examples/policy-simulation — 100 proofs end to end, no API key required.

Full documentation: Policy Simulation · @elsium-ai/observe API · Information-Flow Control · Verifiable Agent Execution

The 0.19.0 release post covers information-flow control and the AI-BOM, which this builds directly on: the proofs it replays and the flow policies it simulates both shipped there.

ElsiumAI is MIT licensed and open source. Created by Eric Utrera (@ebutrera9103).