Each of the N documents has its own independent content-change rate λi and evolves as a real Poisson process — the time between successive real edits is drawn from an exponential distribution with rate λi, exactly as in classical web-crawling freshness models:
next_update = last_update - ln(1 - U)/λi, U ~ Uniform(0,1)
Two crawlers watch the same underlying documents on independent TTL schedules, so the comparison is apples-to-apples: each recrawls document i deterministically every TTLi seconds. The classic Cho & Garcia-Molina "Age" metric measures whether the index copy is behind reality:
age_i(t) = 0 if last_update_i ≤ last_crawl_i
age_i(t) = t - last_update_i otherwise
Given a fixed total crawl budget B (recrawls/s spent across all N documents, i.e. ∑ 1/TTLi = B), two allocation policies are run in parallel on identical documents:
- Uniform — every document gets the same TTL = N/B, ignoring how fast it actually changes.
- Update-rate proportional — each document's recrawl rate is set proportional to its own change rate, ratei = B·λi/∑λ, so TTLi = ∑λ/(B·λi). Volatile documents get short TTLs; static ones get long ones, freeing crawl budget for the documents that actually need it.
Both policies spend the exact same total crawl budget B — the efficiency gain shown is the real, measured drop in mean index age from re-allocating that fixed budget rather than from crawling more.