Toolformer-style self-annotation does not need human labels to decide which tool calls are worth keeping. At each candidate position i in a text, the model samples up to k possible API calls c, executes each one, and inserts the result r before the following tokens y. It then compares the model's own cross-entropy loss on y with and without that result:
ΔL_i = L(y | x) − min_c [ L(y | x, c, r_c) ] − w
keep call i ⇔ max over k samples of ΔL_i > τ
Intuitively: a call is only worth keeping in the training set if plugging its result into the context makes the following tokens measurably easier to predict — enough to outweigh a fixed insertion cost w (the tax of spending extra tokens and latency on a tool call).
- τ (threshold) — how large the loss reduction must be before a call is accepted; raising it makes the filter stricter and shrinks the kept dataset.
- k (candidates per site) — more samples per site raise the chance that at least one candidate call is genuinely useful (an order-statistics effect: the best of k draws).
- Candidate-quality variance — how spread out candidate usefulness is; wider spread means occasional very strong candidates but also many useless ones.
- Insertion-cost weight w — a fixed penalty subtracted from every ΔL, modelling the price of actually calling the tool at inference time.
Top strip: the token stream — candidate sites sprout a blue bar (no-call baseline loss) and a second bar (best call's loss with the result inserted, green if accepted, red if discarded). Drag left/right to pan across the stream. Middle panel: a live histogram of ΔL across this batch's sites, with the τ threshold marked. Bottom panel: acceptance rate over successive batches, so you can watch the filter's yield drift as you retune it or let auto-run keep sampling. This is the same self-supervised filtering step used to build the training data behind tool-augmented language models before any human ever reviews a single example.