Evaluating Agent Revisions as Policies Change

Production agent evaluation must account for changing policies, task-specific outputs, and workflows that can modify external systems. Reindeer evaluates candidate agent revisions against criteria derived from each agent's current specification, cases selected for their feedback value and policy relevance, and replays that isolate production systems from side effects.

Why fixed evaluations are insufficient

Many evaluation methods assume a fixed task, stable labels, and safe replay. Production agents often violate all three assumptions.

Each agent has its own policy, output schema, tools, and business context. A metric that is useful for one workflow may say little about another. The expected behavior also changes when teams revise a playbook or knowledge base, which can make older labels inconsistent with the current policy. Finally, replaying an agent can be unsafe when its tools can send messages, call external APIs, or change customer-facing state.

The Reindeer evaluation stack addresses these conditions through four connected mechanisms.

1. Generate criteria from the agent specification

The criteria-generation workflow reads the agent's current definition, including its system instructions, output schema, and knowledge base. It converts that definition into task-specific, scoreable criteria that are supplied directly to the evaluation judge.

A criterion describes one behavior that the judge can assess, such as whether the agent selected an allowed action, extracted a required field from the available evidence, or followed an escalation rule. Stable criterion keys preserve the identity of a behavioral requirement across revisions, even when its wording changes. This makes it possible to compare regressions by behavior instead of relying only on an aggregate score.

2. Build a feedback-aware golden set

A sample that mirrors production traffic can contain mostly routine cases with little evidence that anyone reviewed the result. Reindeer instead builds golden sets from recent, policy-current cases and samples across several feedback tiers:

Cases with negative feedback that a human later edited
Cases with negative feedback that have not yet been edited
Cases with positive feedback
Cases that were reprocessed
Reviewed cases with no explicit feedback

This sampling method preserves rare corrections and other high-signal cases that raw traffic sampling can bury. Cases that are too recent to have been reviewed are excluded from the no-feedback tier, reducing the amount of unlabeled traffic treated as accepted behavior.

3. Infer the expected result at scoring time

The evaluation workflow derives the reference result from the accepted production case state and its feedback history. When a human supplied negative feedback as a correction, that correction overrides the earlier case state. When feedback is positive or absent, the accepted production state remains the reference.

This approach turns sparse operational feedback into per-case supervision without requiring a separate annotation workflow for every case. It also means that evaluation quality depends on the quality and interpretation of the feedback signal. Teams should review cases where feedback is ambiguous, incomplete, or no longer consistent with the current policy.

4. Replay the candidate revision in eval mode

Reindeer replays a candidate revision on selected cases using the same agent code, model, and tool interfaces used by the production workflow. Eval mode places that execution behind controls that prevent the replay from changing production state or reaching external systems with real credentials.

Credential isolation: Access secrets resolve to deterministic placeholder values, so evaluation runs do not receive production credentials.
Write isolation: Reads can use the permitted reference state, while writes are captured in a per-run overlay rather than committed to production stores.
Side-effect isolation: Read-only and local-only shell operations can run in the evaluation environment. Commands classified as externally side effecting receive synthesized results instead of being executed.

Synthesized results allow the agent to continue along a realistic trajectory after an action such as sending a message, without sending the message or steering the run down an error path caused only by the evaluation environment.

Review evaluation results by criterion

The evaluation output includes verdicts for individual criteria. Teams can use those verdicts to compare revisions, identify cases that remain difficult, and find groups of failures tied to the same policy requirement. Trajectory and end-state differences can also reveal behavior changes that an aggregate score would hide.

These results support several review tasks:

Detect regressions in a specific behavior across revisions.
Find systematic weaknesses that occur across multiple cases.
Identify cases that fall outside the evaluated distribution.
Select additional cases for the next golden-set revision.

Known limitations

No evaluation set provides complete coverage. Cases that were reprocessed several times can have an ambiguous reference state. A correctly absent field can be difficult to distinguish from a missed extraction. Cases without feedback provide weaker evidence of correctness, and the meaning of feedback can drift as reviewers and policies change.

Evaluation results should therefore support revision review rather than replace it. Teams should inspect criterion-level failures, update stale criteria and golden cases when the policy changes, and keep promotion decisions separate from the act of running an evaluation.