Each node is one evaluation trial: a seeded agent run against a scenario suite. Evidence accumulates toward a pass/fail decision every frame, the way a real harness accumulates signal across an agent's tool calls before scoring the episode.
score(t+1) = score(t)·(1-decay) + rate·dt
resolve when score > threshold:
fault_roll < failRate → FAIL (tool injected failure)
adv_roll < redteam·0.35 → INCIDENT (policy violation)
else → PASS
successRate = passes / totalEvaluated
- Eval trials — batch size of concurrently running scenario-suite trials.
- Tool failure injection — probability a resolving trial is forced to FAIL, simulating fault-injected tool harnesses used to test resilience.
- Evaluation throughput — how fast trials accumulate evidence (more tasks/sec through the harness).
- Success threshold — the evidence bar a trial must clear to count as a task success.
- Scenario — Standard runs at nominal rate; Canary throttles throughput for a limited, low-risk rollout; Red-team boosts adversarial pressure, raising the incident (policy-violation) rate.
Real eval pipelines use exactly this layering — scenario suites, canary tests, red-teaming and regression gates — to catch failures and policy violations before an agent ships.