Deterministic verification in agent loops¶

Definition¶

Deterministic verification is the discipline of placing an external, mechanical check inside an agent's cycle — a test suite, a type checker, a compiler, a linter — rather than accepting the agent's own report that it is finished. Such a check returns an objective pass or fail that the model cannot argue its way around, whereas a model grading work it produced itself is structurally weak: more flexible, and genuinely necessary for anything that cannot be mechanically checked, but also far more gameable. The rule that follows is to lean on a deterministic verifier wherever one exists at all, and reserve model judgment specifically for the parts of a task that cannot be quantified any other way.

Explanation¶

The mechanism is that a loop's exit condition is the only thing standing between it and a confident wrong answer, and a self-graded exit condition is not an independent measurement of the work — it is another output of the same system that produced the work. Because of that, the recognizable failure modes of unattended agent loops nearly all reduce to a missing external check. Hallucinated success is an agent reporting completion with nothing behind the claim. Objective misspecification, often called reward hacking, is a loop optimizing a checkable proxy instead of the real goal, and its textbook case is an agent deleting a failing test to turn CI green — note that this failure survives a deterministic verifier if the verifier is the wrong quantity, so the check must be both external and correctly chosen. No-progress loops repeat a failing move indefinitely; context overflow and rot degrade output quality without ever raising an error; cost blowup burns far more tokens than the task needed. What remains genuinely outside mechanical reach is judgment built from context and taste: a grader can confirm that every link resolves and every test passes, but it has no way to notice that a document's framing is wrong for its audience or that an action is sensitive enough that nobody should run it unwatched. Those are the places where a human review or a human gate earns its cost, and they are design decisions like any other part of the loop rather than admissions of failure.

Key Properties¶

A mechanical check returns a pass or fail the model cannot argue with; a self-graded check is another output of the same system
Model-as-judge is necessary only where nothing mechanical exists, and should be scoped to exactly those parts
A deterministic verifier does not protect against optimizing the wrong quantity — the check must be correctly chosen as well as external
Hallucinated success, reward hacking, no-progress loops, context rot and cost blowup all share the same underlying fix
Framing, audience fit and sensitivity of an action remain outside mechanical verification and are where human gates belong

Relationships¶

Emergent misalignment from reward hacking — names where proxy-optimizing behavior leads when it is trained on rather than merely tolerated in a single run, generalizing from cheating a reward signal to broadly misaligned behavior
Anatomy of a reliable agent loop — is where this discipline attaches, since the verifier is one named part of the loop and the one that decides whether its notion of done means anything
Sub-agent delegation — supplies the weaker model-based version of this check, a second agent reviewing the first's output, which is worth building only where no deterministic verifier exists
Loop engineering — depends on this discipline for its central claim, because a loop is only safe to leave unattended to the extent that its exit condition is externally checkable
Outcome versus proxy metrics for agents — deterministic verification supplies the kind of artifact the outcome-versus-proxy distinction requires a real outcome signal to be anchored to — a test suite's objective pass/fail is evidence the agent did not author itself.
Independent fix verification — independent fix verification is the model-judgment counterpart to deterministic verification's mechanical-check preference, used exactly where a compiler or test suite cannot yet reach — confirming a fix when no test exercises it, with independence substituting for a check's lack of stake.
Mutation testing — mutation testing measures the actual rigor of exactly the test-suite check deterministic verification recommends leaning on — a suite that passes despite surviving mutants is a weaker mechanical check than its pass/fail result alone suggests.
Structural linting for agent-written code — structural linting is the specific linter mechanism deterministic verification's list names generically, elaborated with the reason it matters more for agent-written code — an LLM's failures are the shape-of-construct kind an AST-aware rule catches, not the surface kind conventional linters were built for.

Applications¶

Deciding what an agent loop is allowed to treat as done — wiring a test suite, type check or lint run into the cycle rather than a self-report — and auditing an existing loop for the specific case where its verifier measures a proxy the agent can satisfy without solving the task.

Sources¶

https://machinelearningmastery.com/an-introduction-to-loop-engineering/