Outcome versus proxy metrics for agents¶

Definition¶

Outcome versus proxy measurement is the distinction between what falls out of an agent's own record — latency, token and dollar cost, tool-call counts, error counts — and evidence that the work was actually accomplished, which cannot come from the agent at all: a completion claim is produced by the very process whose success is in question, so a real outcome signal has to be anchored to artifacts the agent did not author, such as a test that passed in continuous integration, a pull request that was merged, or a deploy that was not rolled back.

Explanation¶

Proxy metrics are close to free once logging exists at all. A verbose append-only record of every model call with its prompt, response, latency, token count and model version, every tool call with its parameters, result and latency, every error, and one session identifier tying them together yields the entire proxy set with no additional plumbing. Those numbers genuinely catch the obvious pathologies — a session quietly running up a bill, a loop that never terminates because nothing tells it when to stop, a tool failing over and over — and they are worth having for that reason alone. What they cannot do is indicate whether the work was any good, and the trap is subtle because the agent will happily supply the missing signal: an agent whose objective is to finish reports finishing. Self-report is not measurement. Outcome anchors therefore have to be produced by systems outside the agent's authorship and preferably outside its reach, which is why merge status, CI results and rollback status are the usual candidates — they are cheap to read and expensive to fake. Wiring them up is brittle and project-specific, which is exactly why teams stop at proxies; but with no anchor at all the proxy numbers have no direction, and a cheaper, faster, more prolific agent looks like an improvement whether or not it is one. On the underlying data the recommended default is deliberately unfashionable: verbose, append-only, structured as JSON lines, keep everything and trim later, on the grounds that losing the inputs to a session with an unexplained outcome costs far more than the storage bill. The source is a survey newsletter, so the logging minimum and the metric list are practitioner advice rather than measured findings; the epistemic point about self-reported completion stands independently of it.

Key Properties¶

Proxy metrics fall out of the logs for free; outcome evidence must come from artifacts the agent did not produce
A completion claim is generated by the process under evaluation, so it is a claim rather than a measurement
Minimum record: every model call with prompt, response, latency, tokens and version; every tool call with parameters and result; every error; one session id
Proxies reliably catch runaway cost and non-terminating loops, and say nothing about quality
Default to verbose, append-only JSON lines and trim later — losing an unexplained session's inputs costs more than storage

Relationships¶

Agent loop telemetry — supplies the recording layer this distinction consumes and marks its ceiling: a complete span-level account of how a run proceeded still contains no evidence that the run succeeded
Risk-tiered control of agent-authored change — enforces the same rule from the security side, since the acceptance receipt counts as evidence precisely because its contents — build provenance, deterministic check results, a named human approval — originate outside the agent
Loop engineering — depends on this distinction for its stopping condition: an unattended cycle can only be left running if its completion test is mechanically checkable against something other than the agent's own assertion that it is done
Deterministic verification in agent loops — deterministic verification supplies the kind of artifact the outcome-versus-proxy distinction requires a real outcome signal to be anchored to — a test suite's objective pass/fail is evidence the agent did not author itself.

Applications¶

Choosing what to put on an agent dashboard so cost and loop pathologies are visible without mistaking activity for success; wiring one external anchor — merge rate, CI pass rate, rollback rate — before adding more proxy charts; reviewing an agent evaluation that reports task-completion rates to check whether completion was self-declared.

Sources¶

https://newsletter.systemdesign.one/p/agentic-engineering