An evaluation harness pulls tasks from a generator, hands each one to a free agent worker, then checks the worker's output against a hidden test the agent never sees. The scorecard only tallies the oracle's verdict — never the agent's own claim of success.
P(pass) = skill / (skill + strictness)
scoring_time ~ 1 / skill
throughput = workers_busy * skill / mean_scoring_time
- Tasks in flight — how many scenario instances are queued for the harness at once.
- Agent workers — parallel runs of the agent under test, each occupying one evaluation slot.
- Agent skill — how reliably the agent's output would satisfy an average hidden test.
- Test strictness — how tightly the oracle's private checks are set; higher strictness rejects more borderline passes.
Running many seeds through the same harness — instead of trusting one lucky rollout — is what turns a demo into a reproducible scorecard: success rate, cost and reliability, not a single anecdote.